[release 3.2] cherry pick #6223 to release 3.2 (#6234)

cherry pick 6223 to release 3.2

Signed-off-by: dongjiang1989 <dongjiang1989@126.com>

.dockerignore

@@ -1 +1,4 @@
-tmp/
+# exclude all files and folders except bin folder
+!bin
+.idea/
+.vscode/

.gitattributes

@@ -1,2 +1,6 @@
 pkg/cmd/api/spec/api.swagger.json linguist-generated=true
 pkg/cmd/api/spec/static.go linguist-generated=true
+pkg/client/* linguist-generated=true
+config/crds/* linguist-generated=true
+config/rbac/* linguist-generated=true
+zz_generated.deepcopy.go linguist-generated=true

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,41 @@
+---
+name: Bug report
+about: Create a report to help us improve
+---
+
+<!--
+You don't need to remove this comment section, it's invisible on the issues page.
+
+## General remarks
+
+* Attention, please fill out this issues form using English only!
+* 注意！GitHub Issue 仅支持英文，中文 Issue 请在 [论坛](https://kubesphere.com.cn/forum/) 提交。
+* This form is to report bugs. For general usage questions you can join our Slack channel
+       [KubeSphere-users](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE)
+-->
+
+**Describe the Bug**
+A clear and concise description of what the bug is.
+
+For UI issues please also add a screenshot that shows the issue.
+
+**Versions Used**
+KubeSphere:
+Kubernetes: (If KubeSphere installer used, you can skip this)
+
+
+**Environment**
+How many nodes and their hardware configuration:
+
+For example: CentOS 7.5 / 3 masters:  8cpu/8g; 3 nodes: 8cpu/16g
+(and other info are welcomed to help us debugging)
+
+**How To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,69 @@
+---
+name: Feature Request
+about: Have a good idea? Please don't hesitate to write it down, describe the new feature.
+---
+
+<!--
+You don't need to remove this comment section, it's invisible on the issues page.
+
+## General remarks
+
+* Attention, please fill out this issues form using English only!
+* 注意！GitHub Issue 仅支持英文，中文 Issue 请在 [论坛](https://kubesphere.com.cn/forum/) 提交。
+* This form is to report bugs. For general usage questions, you can join our Slack channel
+       [KubeSphere-users](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE)
+-->
+
+**What's it about?**
+<!--
+A clear and concise description of what this feature request is.
+-->
+
+**What's the reason why we need it?**
+<!--
+Please tell us if you think it's a necessary feature for Kubesphere. Give us as many details about it as you can. 
+Two or more use cases might be very helpful when other contributors try to go through this request. If you have some references,
+please just add it below.
+-->
+
+I believe this is an important feature for Kubesphere. There're a few use cases:
+
+* case one
+* case two
+* ...
+
+Please leave your comments below if there's anyone agrees with me. Or just give me a thumb up.
+
+**Area Suggestion**
+<!--
+In order to have a clear issue list, giving an accuracy area is necessary. If you are not sure about it, please just leave it alone.
+
+You can find some possible areas below. Please attention, sometimes crossing multiple areas might be possible. So, you 
+can keep one or more areas in this issue.
+
+> /area alerting
+> /area api
+> /area apiserver
+> /area app-management
+> /area audit
+> /area console
+> /area devops
+> /area documentation
+> /area edge
+> /area iam
+> /area installation
+> /area logging
+> /area microservice
+> /area monitoring
+> /area multicluster
+> /area networking
+> /area notification
+> /area observability
+> /area performance
+> /area security
+> /area storage
+> /area test
+> /area upgrade
+-->
+
+/kind feature-request

.github/ISSUE_TEMPLATE/installation_failure.md

@@ -0,0 +1,30 @@
+---
+name: Installation Issue
+about: Create an issue to help us improve installation
+---
+
+## English only!
+
+**注意！GitHub Issue 仅支持英文，中文 Issue 请在 [论坛](https://kubesphere.com.cn/forum/) 提交。**
+
+**General remarks**
+
+> Please delete this section including header before submitting
+>
+> This form is to report installation issues. For general usage questions you can refer to [KubeSphere Documentation](https://kubesphere.io/docs) or join our Slack channel
+>        [KubeSphere-users](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE)
+
+**What's your question**
+
+
+**Environment: OS & Hardware Information**
+
+> Important: You must describe your environment clearly, e.g. VMware or Bare Metal, CentOS 7.5, 8 C / 16 G (If you install on Linux), Or Kubernetes v1.16 (If you install on K8s).
+
+
+**Error logs or message (Attach logs or screenshot)**
+
+
+**Installer Version**
+
+> e.g. v2.1.0, v2.1.1, v3.0.0

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,62 @@
+<!-- Thanks for sending a pull request! Here are some tips for you:
+
+1. If you want **faster** PR reviews, read how: https://github.com/kubesphere/community/blob/master/developer-guide/development/the-pr-author-guide-to-getting-through-code-review.md
+2. In case you want to know how your PR got reviewed, read: https://github.com/kubesphere/community/blob/master/developer-guide/development/code-review-guide.md
+3. Here are some coding convetions followed by KubeSphere community: https://github.com/kubesphere/community/blob/master/developer-guide/development/coding-conventions.md
+-->
+
+### What type of PR is this?
+<!-- 
+Add one of the following kinds:
+/kind bug
+/kind cleanup
+/kind documentation
+/kind feature
+/kind design
+
+Optionally add one or more of the following kinds if applicable:
+/kind api-change
+/kind deprecation
+/kind failing-test
+/kind flake
+/kind regression
+-->
+
+
+### What this PR does / why we need it:
+
+### Which issue(s) this PR fixes:
+<!--
+Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
+_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
+-->
+Fixes #
+
+### Special notes for reviewers:
+```
+```
+
+### Does this PR introduced a user-facing change?
+<!--
+If no, just write "None" in the release-note block below.
+If yes, a release note is required:
+Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".
+
+For more information on release notes see: https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md
+-->
+```release-note
+
+```
+
+### Additional documentation, usage docs, etc.:
+<!--
+This section can be blank if this pull request does not require a release note.
+Please use the following format for linking documentation or pass the
+section below:
+- [KEP]: <link>
+- [Usage]: <link>
+- [Other doc]: <link>
+-->
+```docs
+
+```

.github/stale.yml

@@ -0,0 +1,20 @@
+# Number of days of inactivity before an issue becomes stale
+daysUntilStale: 90
+# Number of days of inactivity before a stale issue is closed
+daysUntilClose: 30
+# Issues with these labels will never be considered stale
+exemptLabels:
+  - frozen
+  - kind/feature-request
+  - kind/feature
+  - kind/security
+  - kind/design
+staleLabel: stale
+# Comment to post when marking an issue as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed if no further activity occurs. Any further update will
+  cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
+# Comment to post when closing a stale issue. Set to `false` to disable
+closeComment: >
+  This issue is being automatically closed due to inactivity.

.github/workflows/build-multiarch.yaml

@@ -0,0 +1,42 @@
+name: BuildContainerImage
+
+on:
+  push:
+    branches:
+      - 'master'
+      - 'release-*'
+    tags:
+      - 'v*'
+  pull_request:
+    branches:
+      - 'master'
+      - 'release-*'
+
+jobs:
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    env:
+      GO111MODULE: on
+    steps:
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v1
+        with:
+          platforms: all
+
+      - name: Set up Docker buildx
+        uses: docker/setup-buildx-action@v1
+
+      - name: Build and push docker images
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
+        if: github.event_name == 'push'
+        run: |
+          echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
+          REPO=kubespheredev TAG="${GITHUB_REF#refs/*/}" make container-cross-push

.github/workflows/e2e-test.yml

@@ -0,0 +1,44 @@
+name: e2e
+
+on:
+  schedule:
+    # run e2e test every 4 hours
+    - cron: 0 */4 * * *
+  workflow_dispatch:
+jobs:
+  build:
+    name: Test
+    runs-on: ubuntu-latest
+    env:
+      GO111MODULE: on
+    steps:
+
+      - name: Set up Go 1.16
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.16
+        id: go
+
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+
+      - name: Create kind cluster
+        uses: helm/kind-action@v1.2.0
+        with:
+          config: .github/workflows/kind/kind.yaml
+
+      - name: Deploy KubeSphere to Kind
+        run: KIND_CLUSTER_NAME=chart-testing hack/deploy-kubesphere.sh
+
+      - name: Run e2e testing
+        run: go test ./test/e2e
+
+      - name: slack
+        uses: 8398a7/action-slack@v3
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+        with:
+          status: ${{ job.status }}
+          fields: repo,message,commit,author,action,eventName,ref,workflow,job,took
+        if: failure()
+

.github/workflows/kind/kind.yaml

@@ -0,0 +1,11 @@
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+nodes:
+- role: control-plane
+  image: kindest/node:v1.19.7
+  extraMounts:
+  - hostPath: /etc/localtime
+    containerPath: /etc/localtime
+  extraPortMappings:
+  - containerPort: 30881
+    hostPort: 9090

.github/workflows/nightly-builds.yml

@@ -0,0 +1,53 @@
+name: NightlyBuild
+
+on:
+  schedule:
+    # This is a UTC time
+    - cron: "0 16 * * *"
+  # Keep it only for test purpose, comment it once everything is ok
+  workflow_dispatch:
+
+jobs:
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    env:
+      GO111MODULE: on
+    steps:
+
+      - name: Set up Go 1.16
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.16
+        id: go
+
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v1
+        with:
+          platforms: all
+
+      - name: Set up Docker buildx
+        uses: docker/setup-buildx-action@v1
+
+      - name: Build and push docker images
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
+        run: |
+          echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
+          tag=nightly-$(date '+%Y%m%d')
+          REPO=kubespheredev TAG=${tag} make container-cross-push
+
+      - name: slack
+        uses: 8398a7/action-slack@v3
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+        with:
+          status: ${{ job.status }}
+          fields: repo,message,commit,author,action,eventName,ref,workflow,job,took
+        if: failure()

.gitignore

@@ -4,6 +4,7 @@
 *.dll
 *.so
 *.dylib
+*.a
 
 # Test binary, build with `go test -c`
 *.test
@@ -18,8 +19,17 @@ bin/
 
 # Vscode files
 .vscode/
-
-tmp/
+__debug_bin
 
 # OSX trash
 .DS_Store
+api.json
+*.coverprofile
+cover.out
+coverage.txt
+
+kustomize/network/etcd
+apiserver.local.config
+tmp/
+kubesphere.yaml
+testbin/

.travis.yml

@@ -1,32 +0,0 @@
-sudo: required
-
-services:
-  - docker
-
-language: go
-
-go:
-  - 1.10
-  - tip
-
-go_import_path: kubesphere.io/kubesphere
-
-before_install:
-  - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
-  - curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
-  - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
-  - sudo apt-get update
-  - sudo apt-get -y install docker-ce
-  - dep ensure -v
-
-before_script:
-  - docker --version
-
-script:
-  - make fmt-check && make build
-
-deploy:
-  provider: script
-  script: bash install/scripts/docker_push
-  on:
-    branch: master

CONTRIBUTING.md

@@ -0,0 +1,3 @@
+# Guide
+
+This [document](https://github.com/kubesphere/community) walks you through how to get started contributing KubeSphere.

Dockerfile

@@ -1,17 +0,0 @@
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
-# that can be found in the LICENSE file.
-
-FROM kubesphere/kubesphere-builder as builder
-
-WORKDIR /go/src/kubesphere.io/kubesphere/
-COPY . .
-
-RUN go generate kubesphere.io/kubesphere/pkg/version && \
-	go install  kubesphere.io/kubesphere/cmd/...
-
-FROM alpine:3.6
-RUN apk add --update ca-certificates && update-ca-certificates
-COPY --from=builder /go/bin/* /usr/local/bin/
-
-CMD ["sh"]

Dockerfile.dev

@@ -1,10 +0,0 @@
-FROM alpine:3.6
-
-RUN apk add --update ca-certificates \
-    && update-ca-certificates \ 
-    && mkdir -p /etc/kubesphere/ingress-controller
-
-COPY ./bin/* /usr/local/bin/
-COPY ./install/ingress-controller /etc/kubesphere/ingress-controller
-COPY ./install/swagger-ui        /usr/lib/kubesphere/swagger-ui
-CMD ["sh"]

Gopkg.lock

@@ -1,812 +0,0 @@
-# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
-
-
-[[projects]]
-  digest = "1:bf42be3cb1519bf8018dfd99720b1005ee028d947124cab3ccf965da59381df6"
-  name = "github.com/Microsoft/go-winio"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "7da180ee92d8bd8bb8c37fc560e673e6557c392f"
-  version = "v0.4.7"
-
-[[projects]]
-  digest = "1:d1665c44bd5db19aaee18d1b6233c99b0b9a986e8bccb24ef54747547a48027f"
-  name = "github.com/PuerkitoBio/purell"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "0bcb03f4b4d0a9428594752bd2a3b9aa0a9d4bd4"
-  version = "v1.1.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:c739832d67eb1e9cc478a19cc1a1ccd78df0397bf8a32978b759152e205f644b"
-  name = "github.com/PuerkitoBio/urlesc"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "de5bf2ad457846296e2031421a34e2568e304e35"
-
-[[projects]]
-  digest = "1:9e9193aa51197513b3abcb108970d831fbcf40ef96aa845c4f03276e1fa316d2"
-  name = "github.com/Sirupsen/logrus"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "c155da19408a8799da419ed3eeb0cb5db0ad5dbc"
-  version = "v1.0.5"
-
-[[projects]]
-  digest = "1:e49fec8537ec021eeb41d394684bce0365c8db14c8099215f7b509189ddb5852"
-  name = "github.com/antonholmquist/jason"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "c23cef7eaa75a6a5b8810120e167bd590d8fd2ab"
-  version = "v1.0.0"
-
-[[projects]]
-  digest = "1:4fe4dc4ce7ebb5a4b0544c5b411196d23e221800d279a207d76f02812f756c3d"
-  name = "github.com/coreos/etcd"
-  packages = [
-    "auth/authpb",
-    "clientv3",
-    "etcdserver/api/v3rpc/rpctypes",
-    "etcdserver/etcdserverpb",
-    "mvcc/mvccpb",
-    "pkg/tlsutil",
-    "pkg/transport",
-    "pkg/types",
-  ]
-  pruneopts = "UT"
-  revision = "33245c6b5b49130ca99280408fadfab01aac0e48"
-  version = "v3.3.8"
-
-[[projects]]
-  digest = "1:a2c1d0e43bd3baaa071d1b9ed72c27d78169b2b269f71c105ac4ba34b1be4a39"
-  name = "github.com/davecgh/go-spew"
-  packages = ["spew"]
-  pruneopts = "UT"
-  revision = "346938d642f2ec3594ed81d874461961cd0faa76"
-  version = "v1.1.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:4189ee6a3844f555124d9d2656fe7af02fca961c2a9bad9074789df13a0c62e0"
-  name = "github.com/docker/distribution"
-  packages = [
-    "digestset",
-    "reference",
-  ]
-  pruneopts = "UT"
-  revision = "749f6afb4572201e3c37325d0ffedb6f32be8950"
-
-[[projects]]
-  digest = "1:ec821dda59d7dd340498d74f798aa218b2c782bba54a690e866dc4f520d900d5"
-  name = "github.com/docker/docker"
-  packages = [
-    "api",
-    "api/types",
-    "api/types/blkiodev",
-    "api/types/container",
-    "api/types/events",
-    "api/types/filters",
-    "api/types/image",
-    "api/types/mount",
-    "api/types/network",
-    "api/types/registry",
-    "api/types/strslice",
-    "api/types/swarm",
-    "api/types/time",
-    "api/types/versions",
-    "api/types/volume",
-    "client",
-    "pkg/ioutils",
-    "pkg/longpath",
-    "pkg/system",
-    "pkg/tlsconfig",
-  ]
-  pruneopts = "UT"
-  revision = "90d35abf7b3535c1c319c872900fbd76374e521c"
-  version = "v17.05.0-ce-rc3"
-
-[[projects]]
-  branch = "master"
-  digest = "1:811c86996b1ca46729bad2724d4499014c4b9effd05ef8c71b852aad90deb0ce"
-  name = "github.com/docker/go-connections"
-  packages = [
-    "nat",
-    "sockets",
-    "tlsconfig",
-  ]
-  pruneopts = "UT"
-  revision = "7395e3f8aa162843a74ed6d48e79627d9792ac55"
-
-[[projects]]
-  digest = "1:6f82cacd0af5921e99bf3f46748705239b36489464f4529a1589bc895764fb18"
-  name = "github.com/docker/go-units"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "47565b4f722fb6ceae66b95f853feed578a4a51c"
-  version = "v0.3.3"
-
-[[projects]]
-  branch = "master"
-  digest = "1:4841e14252a2cecf11840bd05230412ad469709bbacfc12467e2ce5ad07f339b"
-  name = "github.com/docker/libtrust"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "aabc10ec26b754e797f9028f4589c5b7bd90dc20"
-
-[[projects]]
-  branch = "master"
-  digest = "1:dbb3d1675f5beeb37de6e9b95cc460158ff212902a916e67688b01e0660f41bd"
-  name = "github.com/docker/spdystream"
-  packages = [
-    ".",
-    "spdy",
-  ]
-  pruneopts = "UT"
-  revision = "bc6354cbbc295e925e4c611ffe90c1f287ee54db"
-
-[[projects]]
-  digest = "1:798072bbab2506719d8292cd9b5840a0b5babe0348393bd7097d8fb25ecf0b82"
-  name = "github.com/emicklei/go-restful"
-  packages = [
-    ".",
-    "log",
-  ]
-  pruneopts = "UT"
-  revision = "3658237ded108b4134956c1b3050349d93e7b895"
-  version = "v2.7.1"
-
-[[projects]]
-  digest = "1:e2300c0b15e8b7cb908da64f50e748725c739bcf042a19ceb971680763339888"
-  name = "github.com/emicklei/go-restful-openapi"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "51bf251d405ad1e23511fef0a2dbe40bc70ce2c6"
-  version = "v0.11.0"
-
-[[projects]]
-  digest = "1:2cd7915ab26ede7d95b8749e6b1f933f1c6d5398030684e6505940a10f31cfda"
-  name = "github.com/ghodss/yaml"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "0ca9ea5df5451ffdf184b4428c902747c2c11cd7"
-  version = "v1.0.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:2997679181d901ac8aaf4330d11138ecf3974c6d3334995ff36f20cbd597daf8"
-  name = "github.com/go-openapi/jsonpointer"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "3a0015ad55fa9873f41605d3e8f28cd279c32ab2"
-
-[[projects]]
-  branch = "master"
-  digest = "1:1ae3f233d75a731b164ca9feafd8ed646cbedf1784095876ed6988ce8aa88b1f"
-  name = "github.com/go-openapi/jsonreference"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "3fb327e6747da3043567ee86abd02bb6376b6be2"
-
-[[projects]]
-  branch = "master"
-  digest = "1:cbd9c1cc4ce36075f4ebf0e0525e6cda8597daac1a5eb5f7f88480a3c00e7319"
-  name = "github.com/go-openapi/spec"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "bce47c9386f9ecd6b86f450478a80103c3fe1402"
-
-[[projects]]
-  branch = "master"
-  digest = "1:731022b436cdb9b4b2a53be2ead693467a1474b8b873d4f90cb424fffdc3d0ff"
-  name = "github.com/go-openapi/swag"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "2b0bd4f193d011c203529df626a65d63cb8a79e8"
-
-[[projects]]
-  digest = "1:adea5a94903eb4384abef30f3d878dc9ff6b6b5b0722da25b82e5169216dfb61"
-  name = "github.com/go-sql-driver/mysql"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "d523deb1b23d913de5bdada721a6071e71283618"
-  version = "v1.4.0"
-
-[[projects]]
-  digest = "1:cd559bf134bbedd0dfd5db4d988c88d8f96674fa3f2af0cb5b0dcd5fc0a0a019"
-  name = "github.com/gogo/protobuf"
-  packages = [
-    "gogoproto",
-    "proto",
-    "protoc-gen-gogo/descriptor",
-    "sortkeys",
-  ]
-  pruneopts = "UT"
-  revision = "1adfc126b41513cc696b209667c8656ea7aac67c"
-  version = "v1.0.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:1ba1d79f2810270045c328ae5d674321db34e3aae468eb4233883b473c5c0467"
-  name = "github.com/golang/glog"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "23def4e6c14b4da8ac2ed8007337bc5eb5007998"
-
-[[projects]]
-  digest = "1:17fe264ee908afc795734e8c4e63db2accabaf57326dbf21763a7d6b86096260"
-  name = "github.com/golang/protobuf"
-  packages = [
-    "proto",
-    "ptypes",
-    "ptypes/any",
-    "ptypes/duration",
-    "ptypes/timestamp",
-  ]
-  pruneopts = "UT"
-  revision = "b4deda0973fb4c70b50d226b1af49f3da59f5265"
-  version = "v1.1.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:3ee90c0d94da31b442dde97c99635aaafec68d0b8a3c12ee2075c6bdabeec6bb"
-  name = "github.com/google/gofuzz"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "24818f796faf91cd76ec7bddd72458fbced7a6c1"
-
-[[projects]]
-  digest = "1:65c4414eeb350c47b8de71110150d0ea8a281835b1f386eacaa3ad7325929c21"
-  name = "github.com/googleapis/gnostic"
-  packages = [
-    "OpenAPIv2",
-    "compiler",
-    "extensions",
-  ]
-  pruneopts = "UT"
-  revision = "7c663266750e7d82587642f65e60bc4083f1f84e"
-  version = "v0.2.0"
-
-[[projects]]
-  digest = "1:43dd08a10854b2056e615d1b1d22ac94559d822e1f8b6fcc92c1a1057e85188e"
-  name = "github.com/gorilla/websocket"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "ea4d1f681babbce9545c9c5f3d5194a789c89f5b"
-  version = "v1.2.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:cf296baa185baae04a9a7004efee8511d08e2f5f51d4cbe5375da89722d681db"
-  name = "github.com/hashicorp/golang-lru"
-  packages = [
-    ".",
-    "simplelru",
-  ]
-  pruneopts = "UT"
-  revision = "0fb14efe8c47ae851c0034ed7a448854d3d34cf3"
-
-[[projects]]
-  branch = "master"
-  digest = "1:0778dc7fce1b4669a8bfa7ae506ec1f595b6ab0f8989c1c0d22a8ca1144e9972"
-  name = "github.com/howeyc/gopass"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "bf9dde6d0d2c004a008c27aaee91170c786f6db8"
-
-[[projects]]
-  digest = "1:3e260afa138eab6492b531a3b3d10ab4cb70512d423faa78b8949dec76e66a21"
-  name = "github.com/imdario/mergo"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "9316a62528ac99aaecb4e47eadd6dc8aa6533d58"
-  version = "v0.3.5"
-
-[[projects]]
-  digest = "1:235ae01f32fb5f12c5f6d2e0e05ab48e651ab31c126e45a4efc4f510810941ac"
-  name = "github.com/jinzhu/gorm"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "6ed508ec6a4ecb3531899a69cbc746ccf65a4166"
-  version = "v1.9.1"
-
-[[projects]]
-  branch = "master"
-  digest = "1:fd97437fbb6b7dce04132cf06775bd258cce305c44add58eb55ca86c6c325160"
-  name = "github.com/jinzhu/inflection"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "04140366298a54a039076d798123ffa108fff46c"
-
-[[projects]]
-  digest = "1:b1d4df033414c1a0d85fa7037b9aaf03746314811c860a95ea2d5fd481cd6c35"
-  name = "github.com/json-iterator/go"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "ca39e5af3ece67bbcda3d0f4f56a8e24d9f2dad4"
-  version = "1.1.3"
-
-[[projects]]
-  branch = "master"
-  digest = "1:ada518b8c338e10e0afa443d84671476d3bd1d926e13713938088e8ddbee1a3e"
-  name = "github.com/mailru/easyjson"
-  packages = [
-    "buffer",
-    "jlexer",
-    "jwriter",
-  ]
-  pruneopts = "UT"
-  revision = "3fdea8d05856a0c8df22ed4bc71b3219245e4485"
-
-[[projects]]
-  digest = "1:33422d238f147d247752996a26574ac48dcf472976eda7f5134015f06bf16563"
-  name = "github.com/modern-go/concurrent"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "bacd9c7ef1dd9b15be4a9909b8ac7a4e313eec94"
-  version = "1.0.3"
-
-[[projects]]
-  digest = "1:d711dfcf661439f1ef0b202a02e8a1ff4deac48f26f34253520dcdbecbd7c5f1"
-  name = "github.com/modern-go/reflect2"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "1df9eeb2bb81f327b96228865c5687bc2194af3f"
-  version = "1.0.0"
-
-[[projects]]
-  digest = "1:ee4d4af67d93cc7644157882329023ce9a7bcfce956a079069a9405521c7cc8d"
-  name = "github.com/opencontainers/go-digest"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "279bed98673dd5bef374d3b6e4b09e2af76183bf"
-  version = "v1.0.0-rc1"
-
-[[projects]]
-  digest = "1:40e195917a951a8bf867cd05de2a46aaf1806c50cf92eebf4c16f78cd196f747"
-  name = "github.com/pkg/errors"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "645ef00459ed84a119197bfb8d8205042c6df63d"
-  version = "v0.8.0"
-
-[[projects]]
-  digest = "1:9424f440bba8f7508b69414634aef3b2b3a877e522d8a4624692412805407bb7"
-  name = "github.com/spf13/pflag"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "583c0c0531f06d5278b7d917446061adc344b5cd"
-  version = "v1.0.1"
-
-[[projects]]
-  branch = "master"
-  digest = "1:3f3a05ae0b95893d90b9b3b5afdb79a9b3d96e4e36e099d841ae602e4aca0da8"
-  name = "golang.org/x/crypto"
-  packages = ["ssh/terminal"]
-  pruneopts = "UT"
-  revision = "7f39a6fea4fe9364fb61e1def6a268a51b4f3a06"
-
-[[projects]]
-  branch = "master"
-  digest = "1:bae20a4ea45ad83eb54271a18c820a4ca7c03880aa20d964e2d5bb1d57b1a41a"
-  name = "golang.org/x/net"
-  packages = [
-    "context",
-    "context/ctxhttp",
-    "http/httpguts",
-    "http2",
-    "http2/hpack",
-    "idna",
-    "internal/socks",
-    "internal/timeseries",
-    "proxy",
-    "trace",
-  ]
-  pruneopts = "UT"
-  revision = "db08ff08e8622530d9ed3a0e8ac279f6d4c02196"
-
-[[projects]]
-  branch = "master"
-  digest = "1:a17927b3d78603ae6691d5bf8d3d91467a6edd4ca43c9509347e016a54477f96"
-  name = "golang.org/x/sys"
-  packages = [
-    "unix",
-    "windows",
-  ]
-  pruneopts = "UT"
-  revision = "fc8bd948cf46f9c7af0f07d34151ce25fe90e477"
-
-[[projects]]
-  digest = "1:0c56024909189aee3364b7f21a95a27459f718aa7c199a5c111c36cfffd9eaef"
-  name = "golang.org/x/text"
-  packages = [
-    "collate",
-    "collate/build",
-    "internal/colltab",
-    "internal/gen",
-    "internal/tag",
-    "internal/triegen",
-    "internal/ucd",
-    "language",
-    "secure/bidirule",
-    "transform",
-    "unicode/bidi",
-    "unicode/cldr",
-    "unicode/norm",
-    "unicode/rangetable",
-    "width",
-  ]
-  pruneopts = "UT"
-  revision = "f21a4dfb5e38f5895301dc265a8def02365cc3d0"
-  version = "v0.3.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:c9e7a4b4d47c0ed205d257648b0e5b0440880cb728506e318f8ac7cd36270bc4"
-  name = "golang.org/x/time"
-  packages = ["rate"]
-  pruneopts = "UT"
-  revision = "fbb02b2291d28baffd63558aa44b4b56f178d650"
-
-[[projects]]
-  digest = "1:c25289f43ac4a68d88b02245742347c94f1e108c534dda442188015ff80669b3"
-  name = "google.golang.org/appengine"
-  packages = ["cloudsql"]
-  pruneopts = "UT"
-  revision = "b1f26356af11148e710935ed1ac8a7f5702c7612"
-  version = "v1.1.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:601e63e7d4577f907118bec825902505291918859d223bce015539e79f1160e3"
-  name = "google.golang.org/genproto"
-  packages = ["googleapis/rpc/status"]
-  pruneopts = "UT"
-  revision = "32ee49c4dd805befd833990acba36cb75042378c"
-
-[[projects]]
-  digest = "1:3a98314fd2e43bbd905b33125dad80b10111ba6e5e541db8ed2a953fe01fbb31"
-  name = "google.golang.org/grpc"
-  packages = [
-    ".",
-    "balancer",
-    "balancer/base",
-    "balancer/roundrobin",
-    "codes",
-    "connectivity",
-    "credentials",
-    "encoding",
-    "encoding/proto",
-    "grpclog",
-    "health/grpc_health_v1",
-    "internal",
-    "internal/backoff",
-    "internal/channelz",
-    "internal/grpcrand",
-    "keepalive",
-    "metadata",
-    "naming",
-    "peer",
-    "resolver",
-    "resolver/dns",
-    "resolver/passthrough",
-    "stats",
-    "status",
-    "tap",
-    "transport",
-  ]
-  pruneopts = "UT"
-  revision = "168a6198bcb0ef175f7dacec0b8691fc141dc9b8"
-  version = "v1.13.0"
-
-[[projects]]
-  digest = "1:7a23929a5a0d4266c8f5444dae1e7594dbb0cae1c3091834119b162f81e229ff"
-  name = "gopkg.in/igm/sockjs-go.v2"
-  packages = ["sockjs"]
-  pruneopts = "UT"
-  revision = "d276e9ffe5cc5c271b81198cc77a2adf6c4482d2"
-  version = "v2.0.0"
-
-[[projects]]
-  digest = "1:2d1fbdc6777e5408cabeb02bf336305e724b925ff4546ded0fa8715a7267922a"
-  name = "gopkg.in/inf.v0"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "d2d2541c53f18d2a059457998ce2876cc8e67cbf"
-  version = "v0.9.1"
-
-[[projects]]
-  digest = "1:342378ac4dcb378a5448dd723f0784ae519383532f5e70ade24132c4c8693202"
-  name = "gopkg.in/yaml.v2"
-  packages = ["."]
-  pruneopts = "UT"
-  revision = "5420a8b6744d3b0345ab293f6fcba19c978f1183"
-  version = "v2.2.1"
-
-[[projects]]
-  digest = "1:cae8f1d1d786aa486a7ed236a8c1f099b3b44697ec6bbb5951d7e9bdb53a5125"
-  name = "k8s.io/api"
-  packages = [
-    "admissionregistration/v1alpha1",
-    "admissionregistration/v1beta1",
-    "apps/v1",
-    "apps/v1beta1",
-    "apps/v1beta2",
-    "authentication/v1",
-    "authentication/v1beta1",
-    "authorization/v1",
-    "authorization/v1beta1",
-    "autoscaling/v1",
-    "autoscaling/v2beta1",
-    "batch/v1",
-    "batch/v1beta1",
-    "batch/v2alpha1",
-    "certificates/v1beta1",
-    "core/v1",
-    "events/v1beta1",
-    "extensions/v1beta1",
-    "networking/v1",
-    "policy/v1beta1",
-    "rbac/v1",
-    "rbac/v1alpha1",
-    "rbac/v1beta1",
-    "scheduling/v1alpha1",
-    "settings/v1alpha1",
-    "storage/v1",
-    "storage/v1alpha1",
-    "storage/v1beta1",
-  ]
-  pruneopts = "UT"
-  revision = "73d903622b7391f3312dcbac6483fed484e185f8"
-  version = "kubernetes-1.10.0"
-
-[[projects]]
-  digest = "1:d0089d5f7811ded4279da7a8a66d2721488afec8208d86bdad3f4a20d3687e81"
-  name = "k8s.io/apimachinery"
-  packages = [
-    "pkg/api/errors",
-    "pkg/api/meta",
-    "pkg/api/resource",
-    "pkg/apis/meta/internalversion",
-    "pkg/apis/meta/v1",
-    "pkg/apis/meta/v1/unstructured",
-    "pkg/apis/meta/v1beta1",
-    "pkg/conversion",
-    "pkg/conversion/queryparams",
-    "pkg/fields",
-    "pkg/labels",
-    "pkg/runtime",
-    "pkg/runtime/schema",
-    "pkg/runtime/serializer",
-    "pkg/runtime/serializer/json",
-    "pkg/runtime/serializer/protobuf",
-    "pkg/runtime/serializer/recognizer",
-    "pkg/runtime/serializer/streaming",
-    "pkg/runtime/serializer/versioning",
-    "pkg/selection",
-    "pkg/types",
-    "pkg/util/cache",
-    "pkg/util/clock",
-    "pkg/util/diff",
-    "pkg/util/errors",
-    "pkg/util/framer",
-    "pkg/util/httpstream",
-    "pkg/util/httpstream/spdy",
-    "pkg/util/intstr",
-    "pkg/util/json",
-    "pkg/util/net",
-    "pkg/util/rand",
-    "pkg/util/remotecommand",
-    "pkg/util/runtime",
-    "pkg/util/sets",
-    "pkg/util/validation",
-    "pkg/util/validation/field",
-    "pkg/util/wait",
-    "pkg/util/yaml",
-    "pkg/version",
-    "pkg/watch",
-    "third_party/forked/golang/netutil",
-    "third_party/forked/golang/reflect",
-  ]
-  pruneopts = "UT"
-  revision = "302974c03f7e50f16561ba237db776ab93594ef6"
-  version = "kubernetes-1.10.0"
-
-[[projects]]
-  digest = "1:7ee72261d268f7443085aad95b39fefc17fca826a9bfd8bd2d431bc081852a62"
-  name = "k8s.io/client-go"
-  packages = [
-    "discovery",
-    "informers",
-    "informers/admissionregistration",
-    "informers/admissionregistration/v1alpha1",
-    "informers/admissionregistration/v1beta1",
-    "informers/apps",
-    "informers/apps/v1",
-    "informers/apps/v1beta1",
-    "informers/apps/v1beta2",
-    "informers/autoscaling",
-    "informers/autoscaling/v1",
-    "informers/autoscaling/v2beta1",
-    "informers/batch",
-    "informers/batch/v1",
-    "informers/batch/v1beta1",
-    "informers/batch/v2alpha1",
-    "informers/certificates",
-    "informers/certificates/v1beta1",
-    "informers/core",
-    "informers/core/v1",
-    "informers/events",
-    "informers/events/v1beta1",
-    "informers/extensions",
-    "informers/extensions/v1beta1",
-    "informers/internalinterfaces",
-    "informers/networking",
-    "informers/networking/v1",
-    "informers/policy",
-    "informers/policy/v1beta1",
-    "informers/rbac",
-    "informers/rbac/v1",
-    "informers/rbac/v1alpha1",
-    "informers/rbac/v1beta1",
-    "informers/scheduling",
-    "informers/scheduling/v1alpha1",
-    "informers/settings",
-    "informers/settings/v1alpha1",
-    "informers/storage",
-    "informers/storage/v1",
-    "informers/storage/v1alpha1",
-    "informers/storage/v1beta1",
-    "kubernetes",
-    "kubernetes/scheme",
-    "kubernetes/typed/admissionregistration/v1alpha1",
-    "kubernetes/typed/admissionregistration/v1beta1",
-    "kubernetes/typed/apps/v1",
-    "kubernetes/typed/apps/v1beta1",
-    "kubernetes/typed/apps/v1beta2",
-    "kubernetes/typed/authentication/v1",
-    "kubernetes/typed/authentication/v1beta1",
-    "kubernetes/typed/authorization/v1",
-    "kubernetes/typed/authorization/v1beta1",
-    "kubernetes/typed/autoscaling/v1",
-    "kubernetes/typed/autoscaling/v2beta1",
-    "kubernetes/typed/batch/v1",
-    "kubernetes/typed/batch/v1beta1",
-    "kubernetes/typed/batch/v2alpha1",
-    "kubernetes/typed/certificates/v1beta1",
-    "kubernetes/typed/core/v1",
-    "kubernetes/typed/events/v1beta1",
-    "kubernetes/typed/extensions/v1beta1",
-    "kubernetes/typed/networking/v1",
-    "kubernetes/typed/policy/v1beta1",
-    "kubernetes/typed/rbac/v1",
-    "kubernetes/typed/rbac/v1alpha1",
-    "kubernetes/typed/rbac/v1beta1",
-    "kubernetes/typed/scheduling/v1alpha1",
-    "kubernetes/typed/settings/v1alpha1",
-    "kubernetes/typed/storage/v1",
-    "kubernetes/typed/storage/v1alpha1",
-    "kubernetes/typed/storage/v1beta1",
-    "listers/admissionregistration/v1alpha1",
-    "listers/admissionregistration/v1beta1",
-    "listers/apps/v1",
-    "listers/apps/v1beta1",
-    "listers/apps/v1beta2",
-    "listers/autoscaling/v1",
-    "listers/autoscaling/v2beta1",
-    "listers/batch/v1",
-    "listers/batch/v1beta1",
-    "listers/batch/v2alpha1",
-    "listers/certificates/v1beta1",
-    "listers/core/v1",
-    "listers/events/v1beta1",
-    "listers/extensions/v1beta1",
-    "listers/networking/v1",
-    "listers/policy/v1beta1",
-    "listers/rbac/v1",
-    "listers/rbac/v1alpha1",
-    "listers/rbac/v1beta1",
-    "listers/scheduling/v1alpha1",
-    "listers/settings/v1alpha1",
-    "listers/storage/v1",
-    "listers/storage/v1alpha1",
-    "listers/storage/v1beta1",
-    "pkg/apis/clientauthentication",
-    "pkg/apis/clientauthentication/v1alpha1",
-    "pkg/version",
-    "plugin/pkg/client/auth/exec",
-    "rest",
-    "rest/watch",
-    "tools/auth",
-    "tools/cache",
-    "tools/clientcmd",
-    "tools/clientcmd/api",
-    "tools/clientcmd/api/latest",
-    "tools/clientcmd/api/v1",
-    "tools/metrics",
-    "tools/pager",
-    "tools/reference",
-    "tools/remotecommand",
-    "transport",
-    "transport/spdy",
-    "util/buffer",
-    "util/cert",
-    "util/exec",
-    "util/flowcontrol",
-    "util/homedir",
-    "util/integer",
-    "util/retry",
-  ]
-  pruneopts = "UT"
-  revision = "23781f4d6632d88e869066eaebb743857aa1ef9b"
-  version = "v7.0.0"
-
-[[projects]]
-  digest = "1:2bdbea32607f4effd9e91dadd90baab1ecf224839b613bcaa8f50db5a5f133f5"
-  name = "k8s.io/kubernetes"
-  packages = [
-    "pkg/apis/core",
-    "pkg/util/slice",
-    "pkg/util/version",
-  ]
-  pruneopts = "UT"
-  revision = "5ca598b4ba5abb89bb773071ce452e33fb66339d"
-  version = "v1.10.4"
-
-[solve-meta]
-  analyzer-name = "dep"
-  analyzer-version = 1
-  input-imports = [
-    "github.com/antonholmquist/jason",
-    "github.com/coreos/etcd/clientv3",
-    "github.com/coreos/etcd/etcdserver/api/v3rpc/rpctypes",
-    "github.com/coreos/etcd/pkg/transport",
-    "github.com/docker/docker/api/types",
-    "github.com/docker/docker/client",
-    "github.com/emicklei/go-restful",
-    "github.com/emicklei/go-restful-openapi",
-    "github.com/go-openapi/spec",
-    "github.com/go-sql-driver/mysql",
-    "github.com/golang/glog",
-    "github.com/jinzhu/gorm",
-    "github.com/pkg/errors",
-    "github.com/spf13/pflag",
-    "gopkg.in/igm/sockjs-go.v2/sockjs",
-    "gopkg.in/yaml.v2",
-    "k8s.io/api/apps/v1",
-    "k8s.io/api/apps/v1beta2",
-    "k8s.io/api/autoscaling/v1",
-    "k8s.io/api/batch/v1",
-    "k8s.io/api/batch/v1beta1",
-    "k8s.io/api/core/v1",
-    "k8s.io/api/extensions/v1beta1",
-    "k8s.io/api/policy/v1beta1",
-    "k8s.io/api/rbac/v1",
-    "k8s.io/api/storage/v1",
-    "k8s.io/apimachinery/pkg/api/errors",
-    "k8s.io/apimachinery/pkg/api/resource",
-    "k8s.io/apimachinery/pkg/apis/meta/v1",
-    "k8s.io/apimachinery/pkg/labels",
-    "k8s.io/apimachinery/pkg/types",
-    "k8s.io/apimachinery/pkg/util/sets",
-    "k8s.io/apimachinery/pkg/util/wait",
-    "k8s.io/client-go/informers",
-    "k8s.io/client-go/kubernetes",
-    "k8s.io/client-go/kubernetes/scheme",
-    "k8s.io/client-go/listers/apps/v1",
-    "k8s.io/client-go/listers/batch/v1",
-    "k8s.io/client-go/listers/batch/v1beta1",
-    "k8s.io/client-go/listers/core/v1",
-    "k8s.io/client-go/listers/extensions/v1beta1",
-    "k8s.io/client-go/listers/rbac/v1",
-    "k8s.io/client-go/listers/storage/v1",
-    "k8s.io/client-go/rest",
-    "k8s.io/client-go/tools/cache",
-    "k8s.io/client-go/tools/clientcmd",
-    "k8s.io/client-go/tools/remotecommand",
-    "k8s.io/kubernetes/pkg/apis/core",
-    "k8s.io/kubernetes/pkg/util/slice",
-    "k8s.io/kubernetes/pkg/util/version",
-  ]
-  solver-name = "gps-cdcl"
-  solver-version = 1

Gopkg.toml

@@ -1,90 +0,0 @@
-# Gopkg.toml example
-#
-# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
-# for detailed Gopkg.toml documentation.
-#
-# required = ["github.com/user/thing/cmd/thing"]
-# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
-#
-# [[constraint]]
-#   name = "github.com/user/project"
-#   version = "1.0.0"
-#
-# [[constraint]]
-#   name = "github.com/user/project2"
-#   branch = "dev"
-#   source = "github.com/myfork/project2"
-#
-# [[override]]
-#   name = "github.com/x/y"
-#   version = "2.4.0"
-#
-# [prune]
-#   non-go = false
-#   go-tests = true
-#   unused-packages = true
-
-[[constraint]]
-  name = "github.com/coreos/etcd"
-  version = "3.3.7"
-
-[[constraint]]
-  name = "github.com/docker/docker"
-  version = "v17.05.0-ce"
-
-[[constraint]]
-  name = "github.com/emicklei/go-restful"
-  version = "2.7.1"
-
-[[constraint]]
-  branch = "master"
-  name = "github.com/golang/glog"
-
-[[constraint]]
-  name = "github.com/spf13/pflag"
-  version = "1.0.1"
-
-[[constraint]]
-  name = "gopkg.in/igm/sockjs-go.v2"
-  version = "2.0.0"
-
-[[constraint]]
-  name = "gopkg.in/yaml.v2"
-  version = "2.2.1"
-
-[[constraint]]
-  name = "k8s.io/api"
-  version = "kubernetes-1.10.0"
-
-[[constraint]]
-  name = "k8s.io/apimachinery"
-  version = "kubernetes-1.10.0"
-
-[[constraint]]
-  name = "k8s.io/client-go"
-  version = "7.0.0"
-
-[[constraint]]
-  name = "k8s.io/kubernetes"
-  version = "1.10.4"
-
-[prune]
-  go-tests = true
-  unused-packages = true
-
-# To use reference package:
-#   vendor/github.com/docker/docker/client/container_commit.go:17: undefined: reference.ParseNormalizedNamed
-#   vendor/github.com/docker/docker/client/container_commit.go:25: undefined: reference.TagNameOnly
-#   vendor/github.com/docker/docker/client/container_commit.go:30: undefined: reference.FamiliarNam
-[[override]]
-  name = "github.com/docker/distribution"
-  branch = "master"
-
-# To use reference package:
-#   vendor/github.com/docker/docker/registry/registry.go:30: cannot call non-function tlsconfig.ServerDefault (type tls.Config)
-#   vendor/github.com/docker/docker/registry/registry.go:66: undefined: tlsconfig.SystemCertPool
-#   vendor/github.com/docker/docker/registry/registry.go:168: cannot call non-function tlsconfig.ServerDefault (type tls.Config)
-#   vendor/github.com/docker/docker/registry/service_v2.go:11: cannot call non-function tlsconfig.ServerDefault (type tls.Config)
-[[override]]
-  name = "github.com/docker/go-connections"
-  branch = "master"

Makefile

@@ -2,105 +2,159 @@
 # Use of this source code is governed by a Apache license
 # that can be found in the LICENSE file.
 
-TRAG.Org:=kubesphere
-TRAG.Name:=ks-apiserver
-TRAG.Gopkg:=kubesphere.io/kubesphere
-TRAG.Version:=$(TRAG.Gopkg)/pkg/version
 
-DOCKER_TAGS=latest
-RUN_IN_DOCKER:=docker run -it --rm -v `pwd`:/go/src/$(TRAG.Gopkg) -v `pwd`/tmp/cache:/root/.cache/go-build  -w /go/src/$(TRAG.Gopkg) -e GOBIN=/go/src/$(TRAG.Gopkg)/tmp/bin -e USER_ID=`id -u` -e GROUP_ID=`id -g` kubesphere/kubesphere-builder
-GO_FMT:=goimports -l -w -e -local=kubesphere -srcdir=/go/src/$(TRAG.Gopkg)
-GO_FILES:=./cmd ./pkg
+# Produce CRDs that work back to Kubernetes 1.11 (no version conversion)
+CRD_OPTIONS ?= "crd:trivialVersions=true"
 
-define get_diff_files
-    $(eval DIFF_FILES=$(shell git diff --name-only --diff-filter=ad | grep -E "^(test|cmd|pkg)/.+\.go"))
+GV="network:v1alpha1 servicemesh:v1alpha2 tenant:v1alpha1 tenant:v1alpha2 devops:v1alpha1 iam:v1alpha2 devops:v1alpha3 cluster:v1alpha1 storage:v1alpha1 auditing:v1alpha1 types:v1beta1 quota:v1alpha2 application:v1alpha1 notification:v2beta1"
+
+# Get the currently used golang install path (in GOPATH/bin, unless GOBIN is set)
+ifeq (,$(shell go env GOBIN))
+GOBIN=$(shell go env GOPATH)/bin
+else
+GOBIN=$(shell go env GOBIN)
+endif
+
+OUTPUT_DIR=bin
+ifeq (${GOFLAGS},)
+	# go build with vendor by default.
+	export GOFLAGS=-mod=vendor
+endif
+define ALL_HELP_INFO
+# Build code.
+#
+# Args:
+#   WHAT: Directory names to build.  If any of these directories has a 'main'
+#     package, the build will produce executable files under $(OUT_DIR).
+#     If not specified, "everything" will be built.
+#   GOFLAGS: Extra flags to pass to 'go' when building.
+#   GOLDFLAGS: Extra linking flags passed to 'go' when building.
+#   GOGCFLAGS: Additional go compile flags passed to 'go' when building.
+#
+# Example:
+#   make
+#   make all
+#   make all WHAT=cmd/ks-apiserver
+#     Note: Use the -N -l options to disable compiler optimizations an inlining.
+#           Using these build options allows you to subsequently use source
+#           debugging tools like delve.
 endef
-define get_build_flags
-    $(eval SHORT_VERSION=$(shell git describe --tags --always --dirty="-dev"))
-    $(eval SHA1_VERSION=$(shell git show --quiet --pretty=format:%H))
-	$(eval DATE=$(shell date +'%Y-%m-%dT%H:%M:%S'))
-	$(eval BUILD_FLAG= -X $(TRAG.Version).ShortVersion="$(SHORT_VERSION)" \
-		-X $(TRAG.Version).GitSha1Version="$(SHA1_VERSION)" \
-		-X $(TRAG.Version).BuildDate="$(DATE)")
-endef
-
 .PHONY: all
-all: generate build
+all: test ks-apiserver ks-controller-manager;$(info $(M)...Begin to test and build all of binary.) @ ## Test and build all of binary.
 
-.PHONY: help
 help:
-# TODO: update help info to last version
-	@echo "TODO"
+	@grep -hE '^[ a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
+		awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-17s\033[0m %s\n", $$1, $$2}'
 
-.PHONY: init-vendor
-init-vendor:
-	@if [[ ! -f "$$(which govendor)" ]]; then \
-		go get -u github.com/kardianos/govendor; \
-	fi
-	govendor init
-	govendor add +external
-	@echo "init-vendor done"
+.PHONY: binary
+# Build all of binary
+binary: | ks-apiserver ks-controller-manager; $(info $(M)...Build all of binary.) @ ## Build all of binary.
 
-.PHONY: update-vendor
-update-vendor:
-	@if [[ ! -f "$$(which govendor)" ]]; then \
-		go get -u github.com/kardianos/govendor; \
-	fi
-	govendor update +external
-	govendor list
-	@echo "update-vendor done"
+# Build ks-apiserver binary
+ks-apiserver: ; $(info $(M)...Begin to build ks-apiserver binary.)  @ ## Build ks-apiserver.
+	 hack/gobuild.sh cmd/ks-apiserver; 
 
-.PHONY: update-builder
-update-builder:
-	docker pull kubesphere/kubesphere-builder
-	@echo "update-builder done"
+# Build ks-controller-manager binary
+ks-controller-manager: ; $(info $(M)...Begin to build ks-controller-manager binary.)  @ ## Build ks-controller-manager.
+	hack/gobuild.sh cmd/controller-manager
 
-.PHONY: generate-in-local
-generate-in-local:
-	go generate ./pkg/version/
+# Run all verify scripts hack/verify-*.sh
+verify-all: ; $(info $(M)...Begin to run all verify scripts hack/verify-*.sh.)  @ ## Run all verify scripts hack/verify-*.sh.
+	hack/verify-all.sh
 
-.PHONY: generate
-generate:
-	$(RUN_IN_DOCKER) make generate-in-local
-	@echo "generate done"
+# Build e2e binary
+e2e: ;$(info $(M)...Begin to build e2e binary.)  @ ## Build e2e binary.
+	hack/build_e2e.sh test/e2e
 
-.PHONY: fmt-all
-fmt-all:
-	mkdir -p ./tmp/bin && cp -r ./install ./tmp/
-	$(RUN_IN_DOCKER) $(GO_FMT) $(GO_FILES)
-	@echo "fmt done"
+kind-e2e: ;$(info $(M)...Run e2e test.) @ ## Run e2e test in kind.
+	hack/kind_e2e.sh
 
-.PHONY: fmt
-fmt:
-	$(call get_diff_files)
-	$(if $(DIFF_FILES), \
-		$(RUN_IN_DOCKER) $(GO_FMT) ${DIFF_FILES}, \
-		$(info cannot find modified files from git) \
-	)
-	@echo "fmt done"
+# Run go fmt against code 
+fmt: ;$(info $(M)...Begin to run go fmt against code.)  @ ## Run go fmt against code.
+	gofmt -w ./pkg ./cmd ./tools ./api
 
-.PHONY: fmt-check
-fmt-check: fmt-all
-	$(call get_diff_files)
-	$(if $(DIFF_FILES), \
-		exit 2 \
-	)
+# Format all import, `goimports` is required.
+goimports: ;$(info $(M)...Begin to Format all import.)  @ ## Format all import, `goimports` is required.
+	@hack/update-goimports.sh
 
-.PHONY: build
-build: fmt
-	mkdir -p ./tmp/bin && cp -r ./install/ ./tmp/
-	$(call get_build_flags)
-	$(RUN_IN_DOCKER) time go install -ldflags '$(BUILD_FLAG)' $(TRAG.Gopkg)/cmd/...
-	mv ./tmp/bin/cmd ./tmp/bin/$(TRAG.Name)
-	@docker build -t $(TRAG.Org)/$(TRAG.Name) -f ./Dockerfile.dev ./tmp
-	@docker image prune -f 1>/dev/null 2>&1
-	@echo "build done"
+# Run go vet against code
+vet: ;$(info $(M)...Begin to run go vet against code.)  @ ## Run go vet against code.
+	go vet ./pkg/... ./cmd/...
 
-.PHONY: release
-release:
-	@echo "TODO"
+# Generate manifests e.g. CRD, RBAC etc.
+manifests: ;$(info $(M)...Begin to generate manifests e.g. CRD, RBAC etc..)  @ ## Generate manifests e.g. CRD, RBAC etc.
+	go run ./vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go object:headerFile=./hack/boilerplate.go.txt paths=kubesphere.io/api/application/... rbac:roleName=controller-perms ${CRD_OPTIONS} output:crd:artifacts:config=config/crds
+	go run ./vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go object:headerFile=./hack/boilerplate.go.txt paths=kubesphere.io/api/cluster/... rbac:roleName=controller-perms ${CRD_OPTIONS} output:crd:artifacts:config=config/crds
+	go run ./vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go object:headerFile=./hack/boilerplate.go.txt paths=kubesphere.io/api/devops/... rbac:roleName=controller-perms ${CRD_OPTIONS} output:crd:artifacts:config=config/crds
+	go run ./vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go object:headerFile=./hack/boilerplate.go.txt paths=kubesphere.io/api/iam/... rbac:roleName=controller-perms ${CRD_OPTIONS} output:crd:artifacts:config=config/crds
+	go run ./vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go object:headerFile=./hack/boilerplate.go.txt paths=kubesphere.io/api/network/v1alpha1/... rbac:roleName=controller-perms ${CRD_OPTIONS} output:crd:artifacts:config=config/crds
+	go run ./vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go object:headerFile=./hack/boilerplate.go.txt paths=kubesphere.io/api/quota/... rbac:roleName=controller-perms ${CRD_OPTIONS} output:crd:artifacts:config=config/crds
+	go run ./vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go object:headerFile=./hack/boilerplate.go.txt paths=kubesphere.io/api/storage/... rbac:roleName=controller-perms ${CRD_OPTIONS} output:crd:artifacts:config=config/crds
+	go run ./vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go object:headerFile=./hack/boilerplate.go.txt paths=kubesphere.io/api/tenant/... rbac:roleName=controller-perms ${CRD_OPTIONS} output:crd:artifacts:config=config/crds
+
+deploy: manifests ;$(info $(M)...Begin to deploy.)  @ ## Deploy.
+	kubectl apply -f config/crds
+	kustomize build config/default | kubectl apply -f -
+
+mockgen: ;$(info $(M)...Begin to mockgen.)  @ ## Mockgen.
+	mockgen -package=openpitrix -source=pkg/simple/client/openpitrix/openpitrix.go -destination=pkg/simple/client/openpitrix/mock.go
+
+deepcopy: ;$(info $(M)...Begin to deepcopy.)  @ ## Deepcopy.
+	hack/generate_group.sh "deepcopy" kubesphere.io/api kubesphere.io/api ${GV} --output-base=staging/src/  -h "hack/boilerplate.go.txt"
+
+openapi: ;$(info $(M)...Begin to openapi.)  @ ## Openapi.
+	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./vendor/kubesphere.io/api/tenant/v1alpha1 -p kubesphere.io/api/tenant/v1alpha1 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list  --output-base=staging/src/
+	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./vendor/kubesphere.io/api/network/v1alpha1 -p kubesphere.io/api/network/v1alpha1 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list  --output-base=staging/src/
+	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./vendor/kubesphere.io/api/servicemesh/v1alpha2 -p kubesphere.io/api/servicemesh/v1alpha2 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list  --output-base=staging/src/
+	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/api/networking/v1,./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./vendor/k8s.io/apimachinery/pkg/util/intstr,./vendor/kubesphere.io/api/network/v1alpha1 -p kubesphere.io/api/network/v1alpha1 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list  --output-base=staging/src/
+	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./vendor/kubesphere.io/api/devops/v1alpha1,./vendor/k8s.io/apimachinery/pkg/runtime,./vendor/k8s.io/api/core/v1 -p kubesphere.io/api/devops/v1alpha1 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list  --output-base=staging/src/
+	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./vendor/kubesphere.io/api/cluster/v1alpha1,./vendor/k8s.io/apimachinery/pkg/runtime,./vendor/k8s.io/api/core/v1 -p kubesphere.io/api/cluster/v1alpha1 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list  --output-base=staging/src/
+	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./vendor/kubesphere.io/api/devops/v1alpha3,./vendor/k8s.io/apimachinery/pkg/runtime -p kubesphere.io/api/devops/v1alpha3 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list  --output-base=staging/src/
+	go run ./tools/cmd/crd-doc-gen/main.go
+	go run ./tools/cmd/doc-gen/main.go
+
+container: ;$(info $(M)...Begin to build the docker image.)  @ ## Build the docker image.
+	DRY_RUN=true hack/docker_build.sh
+
+container-push: ;$(info $(M)...Begin to build and push.)  @ ## Build and Push.
+	hack/docker_build.sh
+
+container-cross: ; $(info $(M)...Begin to build container images for multiple platforms.)  @ ## Build container images for multiple platforms. Currently, only linux/amd64,linux/arm64 are supported.
+	DRY_RUN=true hack/docker_build_multiarch.sh
+
+container-cross-push: ; $(info $(M)...Begin to build and push.)  @ ## Build and Push.
+	hack/docker_build_multiarch.sh
+
+helm-package: ; $(info $(M)...Begin to helm-package.)  @ ## Helm-package.
+	ls config/crds/ | xargs -i cp -r config/crds/{} config/ks-core/crds/
+	helm package config/ks-core --app-version=v3.1.0 --version=0.1.0 -d ./bin
+
+helm-deploy: ; $(info $(M)...Begin to helm-deploy.)  @ ## Helm-deploy.
+	ls config/crds/ | xargs -i cp -r config/crds/{} config/ks-core/crds/
+	- kubectl create ns kubesphere-controls-system
+	helm upgrade --install ks-core ./config/ks-core -n kubesphere-system --create-namespace
+	kubectl apply -f https://raw.githubusercontent.com/kubesphere/ks-installer/master/roles/ks-core/prepare/files/ks-init/role-templates.yaml
+
+helm-uninstall: ; $(info $(M)...Begin to helm-uninstall.)  @ ## Helm-uninstall.
+	- kubectl delete ns kubesphere-controls-system
+	helm uninstall ks-core -n kubesphere-system
+	kubectl delete -f https://raw.githubusercontent.com/kubesphere/ks-installer/master/roles/ks-core/prepare/files/ks-init/role-templates.yaml
+
+# Run tests
+ENVTEST_ASSETS_DIR=$(shell pwd)/testbin
+test: vet test-env ;$(info $(M)...Begin to run tests.)  @ ## Run tests.
+	export KUBEBUILDER_ASSETS=$(shell pwd)/testbin/bin; go test ./pkg/... ./cmd/... -covermode=atomic -coverprofile=coverage.txt
+	cd staging/src/kubesphere.io/api ; GOFLAGS="" go test ./...
+	cd staging/src/kubesphere.io/client-go ; GOFLAGS="" go test ./...
+
+.PHONY: test-env
+test-env: ;$(info $(M)...Begin to setup test env) @ ## Download unit test libraries e.g. kube-apiserver etcd.
+	@hack/setup-kubebuilder-env.sh
 
 .PHONY: clean
-clean:
+clean: ;$(info $(M)...Begin to clean.)  @ ## Clean.
 	-make -C ./pkg/version clean
 	@echo "ok"
+
+clientset:  ;$(info $(M)...Begin to find or download controller-gen.)  @ ## Find or download controller-gen,download controller-gen if necessary.
+	./hack/generate_client.sh ${GV}

OWNERS

@@ -0,0 +1,26 @@
+approvers:
+    - zryfish #oncall
+    - rayzhou2017
+    - wansir
+    - zheng1
+    - benjaminhuo
+
+reviewers:
+    - rayzhou2017
+    - zryfish
+    - benjaminhuo
+    - calvinyv
+    - FeynmanZhou
+    - pixiake
+    - wansir
+    - zheng1
+    - stoneshi-yunify
+    - linuxsuren
+    - RolandMa1986
+    - wanjunlei
+    - xyz-li
+    - junotx
+    - yuswift
+    - zhu733756
+    - JohnNiang
+    - dkeven

PROJECT

@@ -0,0 +1,3 @@
+version: "1"
+domain: kubesphere.io
+repo: kubesphere.io/kubesphere

README.md

@@ -1,37 +1,176 @@
-# KubeSphere
-[![License](http://img.shields.io/badge/license-apache%20v2-blue.svg)](https://github.com/KubeSphere/KubeSphere/blob/master/LICENSE)
-[![Build Status](https://travis-ci.org/kubesphere/kubesphere.svg?branch=master)](https://travis-ci.org/kubesphere/kubesphere)
+<p align="center">
+<img src="docs/images/kubesphere-logo.png" alt="banner" width="200px">
+</p>
+
+<p align="center">
+<b>The container platform tailored for <i>Kubernetes multi-cloud, datacenter, and edge</i> management</b>
+</p>
+
+<p align=center>
+<a href="https://goreportcard.com/report/github.com/kubesphere/kubesphere"><img src="https://goreportcard.com/badge/github.com/kubesphere/kubesphere" alt="A+"></a>
+<a href="https://hub.docker.com/r/kubesphere/ks-installer"><img src="https://img.shields.io/docker/pulls/kubesphere/ks-installer"></a>
+<a href="https://github.com/search?q=user%3Akubesphere+user%3Akubesphere-sigs+label%3A%22good+first+issue%22+state%3Aopen&type=Issues&ref=advsearch&l=&l="><img src="https://img.shields.io/github/issues/badges/shields/good%20first%20issue" alt="good first"></a>
+<a href="https://twitter.com/intent/follow?screen_name=KubeSphere"><img src="https://img.shields.io/twitter/follow/KubeSphere?style=social" alt="follow on Twitter"></a>
+<a href="https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE"><img src="https://img.shields.io/badge/Slack-600%2B-blueviolet?logo=slack&amp;logoColor=white"></a>
+<a href="https://www.youtube.com/channel/UCyTdUQUYjf7XLjxECx63Hpw"><img src="https://img.shields.io/youtube/channel/subscribers/UCyTdUQUYjf7XLjxECx63Hpw?style=social"></a>
+</p>
+
 
 ----
-***KubeSphere*** is a distribution of [Kubernetes](https://kubernetes.io), aimed to provide quick setup, friendly and easily use, and powerful management features for Kubernetes clusters, which could help both personal and enterprise users, reduce their learning curve of Kubernetes, accelerate their transform process from other container platforms to Kubernetes.   
 
-**Features:**
- - Multiple IaaS platform support, including baremetal/KVM/QingCloud, and more will be supported in future release.  
- - Easy setup of Kubernetes standalone(only one master node) and cluster environment(including High Availability support).  
- - Powerful management console to help business users to manage and monitor the Kubernetes.  
- - Integrate with [OpenPitrix](https://github.com/openpitrix) to provide full life cycle of application management and be compatible of helm package.  
- - Support popular open source network solutions, including calico and flannel, also could use [qingcloud hostnic solution](https://github.com/yunify/hostnic-cni) if the Kubernetes is deployed on QingCloud platform.  
- - Support popular open source storage solutions, including Glusterfs and Cephfs, also could use [qingcloud storage solution](https://github.com/yunify/qingcloud-csi) or [qingstor storage solution](https://github.com/yunify/qingstor-csi) if the Kubernetes is deployed on QingCloud platform or QingStor NeonSAN.  
- - CI/CD support.  
- - Service Mesh support.  
- - Multiple image registries support.  
- - Integrate with QingCloud IAM.  
+## What is KubeSphere
+
+> English | [中文](README_zh.md)
+
+[KubeSphere](https://kubesphere.io/) is a **distributed operating system for cloud-native application management**, using [Kubernetes](https://kubernetes.io) as its kernel. It provides a plug-and-play architecture, allowing third-party applications to be seamlessly integrated into its ecosystem. KubeSphere is also a multi-tenant container platform with full-stack automated IT operation and streamlined DevOps workflows. It provides developer-friendly wizard web UI, helping enterprises to build out a more robust and feature-rich platform, which includes most common functionalities needed for enterprise Kubernetes strategy, see [Feature List](#features) for details.
+
+The following screenshots give a close insight into KubeSphere. Please check [What is KubeSphere](https://kubesphere.io/docs/introduction/what-is-kubesphere/) for further information.
+
+<table>
+  <tr>
+      <td width="50%" align="center"><b>Workbench</b></td>
+      <td width="50%" align="center"><b>Project Resources</b></td>
+  </tr>
+  <tr>
+     <td><img src="docs/images/console.png"/></td>
+     <td><img src="docs/images/project.png"/></td>
+  </tr>
+  <tr>
+      <td width="50%" align="center"><b>CI/CD Pipeline</b></td>
+      <td width="50%" align="center"><b>App Store</b></td>
+  </tr>
+  <tr>
+     <td><img src="docs/images/cicd.png"/></td>
+     <td><img src="docs/images/app-store.png"/></td>
+  </tr>
+</table>
+
+## Demo Environment
+
+🎮 Using the account `demo1 / Demo123` to log in the [demo environment](https://demo.kubesphere.io/). Please note the account is granted view access. 
+
+🖥 You can also have a quick view of [Demo video](https://youtu.be/YxZ1YUv0CYs).
+
+## Features
+
+<details>
+  <summary><b>🕸 Provisioning Kubernetes Cluster</b></summary>
+  Support deploy Kubernetes on any infrastructure, support online and air-gapped installation, <a href="https://kubesphere.io/docs/installing-on-linux/introduction/intro/">learn more</a>.
+  </details>
+
+<details>
+  <summary><b>🔗 Kubernetes Multi-cluster Management</b></summary>
+  Provide a centralized control plane to manage multiple Kubernetes clusters, support propagate an app to multiple K8s clusters across different cloud providers.
+  </details>
+
+<details>
+  <summary><b>🤖 Kubernetes DevOps</b></summary>
+  Provide out-of-box CI/CD based on Jenkins, and offers automated workflow tools including binary-to-image (B2I) and source-to-image (S2I), <a href="https://kubesphere.io/devops/">learn more</a>.
+  </details>
+
+<details>
+  <summary><b>🔎 Cloud Native Observability</b></summary>
+  Multi-dimensional monitoring, events and auditing logs are supported; multi-tenant log query and collection, alerting and notification are built-in, <a href="https://kubesphere.io/observability/">learn more</a>.
+  </details>
+
+<details>
+  <summary><b>🧩 Service Mesh (Istio-based)</b></summary>
+  Provide fine-grained traffic management, observability and tracing for distributed microservice applications, provides visualization for traffic topology, <a href="https://kubesphere.io/service-mesh/">learn more</a>.
+  </details>
+
+<details>
+  <summary><b>💻 App Store</b></summary>
+  Provide an App Store for Helm-based applications, and offer application lifecycle management on Kubernetes platform, <a href="https://kubesphere.io/docs/pluggable-components/app-store/">learn more</a>.
+  </details>
+
+<details>
+  <summary><b>💡 Edge Computing Platform</b></summary>
+  KubeSphere integrates <a href="https://kubeedge.io/en/">KubeEdge</a> to enable users to deploy applications on the edge devices and view logs and monitoring metrics of them on the console, <a href="https://kubesphere.io/docs/pluggable-components/kubeedge/">learn more</a>.
+  </details>
+
+<details>
+  <summary><b>📊 Metering and Billing</b></summary>
+  Track resource consumption at different levels on a unified dashboard, which helps you make better-informed decisions on planning and reduce the cost, <a href="https://kubesphere.io/docs/toolbox/metering-and-billing/view-resource-consumption/">learn more</a>.
+  </details>
+
+<details>
+  <summary><b>🗃 Support Multiple Storage and Networking Solutions</b></summary>
+  <li>Support GlusterFS, CephRBD, NFS, LocalPV solutions, and provide CSI plugins to consume storage from multiple cloud providers.</li><li>Provide Load Balancer Implementation <a href="https://github.com/kubesphere/openelb">OpenELB</a> for Kubernetes in bare-metal, edge, and virtualization.</li><li> Provides network policy and Pod IP pools management, support Calico, Flannel, Kube-OVN</li>.</li>.
+  </details>
+
+<details>
+  <summary><b>🏘 Multi-tenancy</b></summary>
+  Provide unified authentication with fine-grained roles and three-tier authorization system, and support AD/LDAP authentication.
+  </details>
+
+## Architecture
+
+KubeSphere uses a loosely-coupled architecture that separates the [frontend](https://github.com/kubesphere/console) from the [backend](https://github.com/kubesphere/kubesphere). External systems can access the components of the backend through the REST APIs. 
+
+![Architecture](docs/images/architecture.png)
+
 ----
 
-## Motivation
+## Latest Release
 
-The project originates from the requirement and pains we heard from our customers on public and private QingCloud platform, who have strong will to deploy Kubernetes in their IT system but struggle on completed setup process and long learning curve. With help of KubeSphere, their IT operators could setup Kubernetes environment quickly and use an easy management UI interface to mange their applications, also KubeSphere provides more features to help customers to handle daily business more easily, including CI/CD, micro services management...etc.
+🎉 KubeSphere 3.1.1 is now available! See the [Release Notes For 3.1.1](https://kubesphere.io/docs/release/release-v311/) for the updates.
 
-Getting Started
----------------
-**TBD**
+## Installation
 
-## Design
+KubeSphere can run anywhere from on-premise datacenter to any cloud to edge. In addition, it can be deployed on any version-compatible Kubernetes cluster.
 
-## Contributing to the project
+### Quick Start
 
-All members of the KubeSphere community must abide by [Code of Conduct](docs/code-of-conduct.md). Only by respecting each other can we develop a productive, collaborative community.
+1. Run the following commands to install KubeSphere on an exiting Kubernetes cluster:
 
-You can then find out more detail [here](docs/welcome-toKubeSphere-new-developer-guide.md).
+```yaml
+kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.1.1/kubesphere-installer.yaml
+   
+kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.1.1/cluster-configuration.yaml
+```
 
+2. You can run the following command to view the installation logs. After KubeSphere is successfully installed, you can use `http://IP:30880` to access the KubeSphere Console with the default account and password (admin/P@88w0rd).
 
+```yaml
+kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
+```
+
+> 👨‍💻 No Kubernetes cluster? Try [All-in-one](https://kubesphere.io/docs/quick-start/all-in-one-on-linux/) to install a single-node Kubernetes and KubeSphere on your Linux machine. 
+
+### KubeSphere for hosted Kubernetes services
+
+KubeSphere is hosted on the following cloud providers, you can try KubeSphere by one-click installation on their hosted Kubernetes services. 
+
+- [KubeSphere for Amazon EKS](https://aws.amazon.com/quickstart/architecture/qingcloud-kubesphere/)
+- [KubeSphere for Azure AKS](https://market.azure.cn/marketplace/apps/qingcloud.kubesphere)
+- [KubeSphere for DigitalOcean Kubernetes](https://marketplace.digitalocean.com/apps/kubesphere)
+- [KubeSphere on QingCloud AppCenter(QKE)](https://www.qingcloud.com/products/kubesphereqke)
+
+You can also install KubeSphere on other hosted Kubernetes services within minutes, see the [step-by-step guides](https://kubesphere.io/docs/installing-on-kubernetes/) to get started.
+
+> 👨‍💻 No internet access? Refer to the [Air-gapped Installation on Kubernetes](https://kubesphere.io/docs/installing-on-kubernetes/on-prem-kubernetes/install-ks-on-linux-airgapped/) or [Air-gapped Installation on Linux](https://kubesphere.io/docs/installing-on-linux/introduction/air-gapped-installation/) for instructions on how to use private registry to install KubeSphere.
+
+## Contributing, Support, Discussion, and Community
+
+We :heart: your contribution. The [community](https://github.com/kubesphere/community) walks you through how to get started contributing KubeSphere. The [development guide](https://github.com/kubesphere/community/tree/master/developer-guide/development) explains how to set up development environment.
+
+- [Slack Channel](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE)
+- [Youtube](https://www.youtube.com/channel/UCyTdUQUYjf7XLjxECx63Hpw)
+- [Follow us on Twitter](https://twitter.com/KubeSphere)
+
+Please submit any KubeSphere bugs, issues, and feature requests to [KubeSphere GitHub Issue](https://github.com/kubesphere/kubesphere/issues).
+
+## Who are using KubeSphere
+
+The [user case studies](https://kubesphere.io/case/) page includes the user list of the project. You can [leave a comment](https://github.com/kubesphere/kubesphere/issues/4123) to let us know your use case.
+
+## Landscapes
+
+<p align="center">
+<br/><br/>
+<img src="https://landscape.cncf.io/images/left-logo.svg" width="150"/>&nbsp;&nbsp;<img src="https://landscape.cncf.io/images/right-logo.svg" width="200"/>&nbsp;&nbsp;
+<br/><br/>
+KubeSphere is a member of CNCF and a <a href="https://www.cncf.io/certification/software-conformance/#logos">Kubernetes Conformance Certified platform
+</a>, which enriches the <a href="https://landscape.cncf.io/?landscape=observability-and-analysis&license=apache-license-2-0">CNCF CLOUD NATIVE Landscape.
+</a>
+</p>

README_zh.md

@@ -0,0 +1,178 @@
+<p align="center">
+<img src="docs/images/kubesphere-logo.png" alt="banner" width="200px">
+</p>
+
+<p align="center">
+<b>为<i> Kubernetes 多云、数据中心和边缘 </i>管理而定制的容器平台</b>
+</p>
+
+<p align=center>
+<a href="https://goreportcard.com/report/github.com/kubesphere/kubesphere"><img src="https://goreportcard.com/badge/github.com/kubesphere/kubesphere" alt="A+"></a>
+<a href="https://hub.docker.com/r/kubesphere/ks-installer"><img src="https://img.shields.io/docker/pulls/kubesphere/ks-installer"></a>
+<a href="https://github.com/search?q=user%3Akubesphere+user%3Akubesphere-sigs+label%3A%22good+first+issue%22+state%3Aopen&type=Issues&ref=advsearch&l=&l="><img src="https://img.shields.io/github/issues/badges/shields/good%20first%20issue" alt="good first"></a>
+<a href="https://twitter.com/intent/follow?screen_name=KubeSphere"><img src="https://img.shields.io/twitter/follow/KubeSphere?style=social" alt="follow on Twitter"></a>
+<a href="https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE"><img src="https://img.shields.io/badge/Slack-600%2B-blueviolet?logo=slack&amp;logoColor=white"></a>
+<a href="https://www.youtube.com/channel/UCyTdUQUYjf7XLjxECx63Hpw"><img src="https://img.shields.io/youtube/channel/subscribers/UCyTdUQUYjf7XLjxECx63Hpw?style=social"></a>
+</p>
+
+
+----
+
+## KubeSphere 是什么
+
+> [English](README.md) | 中文
+
+[KubeSphere](https://kubesphere.io/zh/) 愿景是打造一个以 [Kubernetes](https://kubernetes.io/zh/) 为内核的 **云原生分布式操作系统**，它的架构可以非常方便地使第三方应用与云原生生态组件进行即插即用（plug-and-play）的集成，支持云原生应用在多云与多集群的统一分发和运维管理。 KubeSphere 也是一个多租户容器平台，提供全栈的 IT 自动化运维的能力，简化企业的 DevOps 工作流。KubeSphere 提供了运维友好的向导式操作界面，帮助企业快速构建一个强大和功能丰富的容器云平台，详情请参阅 [平台功能](#平台功能) 。
+
+下面的屏幕截图让我们进一步了解 KubeSphere，关于 KubeSphere 更详细的介绍与说明请参阅 [什么是 KubeSphere](https://kubesphere.io/zh/docs/introduction/what-is-kubesphere/) 。
+
+<table>
+  <tr>
+      <td width="50%" align="center"><b>工作台</b></td>
+      <td width="50%" align="center"><b>项目资源</b></td>
+  </tr>
+  <tr>
+     <td><img src="docs/images/console.png"/></td>
+     <td><img src="docs/images/project.png"/></td>
+  </tr>
+  <tr>
+      <td width="50%" align="center"><b>CI/CD 流水线</b></td>
+      <td width="50%" align="center"><b>应用商店</b></td>
+  </tr>
+  <tr>
+     <td><img src="docs/images/cicd.png"/></td>
+     <td><img src="docs/images/app-store.png"/></td>
+  </tr>
+</table>
+
+## Demo 环境
+
+🎮 使用账号 `demo1 / Demo123` 登录 [Demo 环境](https://demo.kubesphere.io/) 。请注意，该帐户仅授予了 view 权限。
+
+🖥 您还可以快速查看[Demo 视频](https://youtu.be/YxZ1YUv0CYs) 。
+
+## 平台功能
+
+<details>
+  <summary><b>🕸 部署 Kubernetes 集群</b></summary>
+  支持在任何基础设施上部署 Kubernetes，支持在线安装和离线安装，<a href="https://kubesphere.io/zh/docs/installing-on-linux/introduction/intro/">了解更多</a> 。
+  </details>
+
+<details>
+  <summary><b>🔗 Kubernetes 多集群管理</b></summary>
+  提供集中控制平台来管理多个 Kubernetes 集群，支持将应用程序发布到跨不同云供应商的多个k8集群上。
+  </details>
+
+<details>
+  <summary><b>🤖 Kubernetes DevOps</b></summary>
+  提供开箱即用的基于 Jenkins 的 CI/CD，并内置自动化流水线插件，包括Binary-to-Image (B2I) 和Source-to-Image (S2I)，<a href="https://kubesphere.io/zh/devops/">了解更多</a> 。
+  </details>
+
+<details>
+  <summary><b>🔎 云原生可观测性</b></summary>
+  支持多维度监控、事件和审计日志；内置多租户日志查询和收集，告警和通知，<a href="https://kubesphere.io/zh/observability/">了解更多</a> 。
+  </details>
+
+<details>
+  <summary><b>🧩 基于 Istio 的微服务治理</b></summary>
+  为分布式微服务应用程序提供细粒度的流量管理、可观测性和服务跟踪，支持可视化的流量拓扑，<a href="https://kubesphere.io/zh/service-mesh/">了解更多</a> 。
+  </details>
+
+<details>
+  <summary><b>💻 应用商店</b></summary>
+  为基于 Helm 的应用程序提供应用商店，并在 Kubernetes 平台上提供应用程序生命周期管理功能，<a href="https://kubesphere.io/zh/docs/pluggable-components/app-store/">了解更多</a> 。
+  </details>
+
+<details>
+  <summary><b>💡 Kubernetes 边缘节点管理</b></summary>
+  基于 <a href="https://kubeedge.io/zh/">KubeEdge</a> 实现应用与工作负载在云端与边缘节点的统一分发与管理，解决在海量边、端设备上完成应用交付、运维、管控的需求，<a href= "https://kubesphere.io/zh/docs/pluggable-components/kubeedge/">了解更多</a> 。
+  </details>
+
+<details>
+  <summary><b>📊 多维度计量与计费</b></summary>
+  提供基于集群与租户的多维度资源计量与计费的监控报表，让 Kubernetes 运营成本更透明，<a href="https://kubesphere.io/zh/docs/toolbox/metering-and-billing/view-resource-consumption/">了解更多</a> 。
+  </details>
+
+<details>
+  <summary><b>🗃 支持多种存储和网络解决方案</b></summary>
+  <li>支持 GlusterFS、CephRBD、NFS、LocalPV ，并提供多个 CSI 插件对接公有云与企业级存储。</li><li>提供Kubernetes在裸机、边缘和虚拟化中的负载均衡器实现 <a href="https://github.com/kubesphere/openelb">OpenELB</a> 。</li><li>提供网络策略和容器组 IP 池管理，支持 Calico、Flannel、Kube-OVN。</li>
+  </details>
+
+<details>
+  <summary><b>🏘 多租户</b></summary>
+  提供统一的认证鉴权与细粒度的基于角色的授权系统，支持对接 AD/LDAP 。
+  </details>
+
+## 架构说明
+
+KubeSphere 使用前后端分离的架构，将 [前端](https://github.com/kubesphere/console) 与 [后端](https://github.com/kubesphere/kubesphere) 分开。后端的各个功能组件可通过 REST API 对接外部系统。
+
+![Architecture](docs/images/architecture.png)
+
+----
+
+## 最新版本
+
+🎉 KubeSphere 3.1.1 全新发布！相关更新信息，请参阅 [Release Notes For 3.1.1](https://kubesphere.io/zh/docs/release/release-v311/) 。
+
+## 安装
+
+KubeSphere 支持在任意平台运行，从本地数据中心到混合多云再走向边缘。此外，KubeSphere 可以部署在任何版本兼容的 Kubernetes 集群上。
+
+### 快速入门
+
+1. 运行以下命令以在现有 Kubernetes 集群上安装 KubeSphere：
+
+```yaml
+kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.1.1/kubesphere-installer.yaml
+   
+kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.1.1/cluster-configuration.yaml
+```
+
+2. 您可以运行以下命令查看安装日志。 KubeSphere 安装成功后，您可以使用`http://IP:30880` 以默认账号和密码（admin/P@88w0rd）访问KubeSphere 控制台。
+
+```yaml
+kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
+```
+
+> 👨‍💻 没有 Kubernetes 集群? 可以尝试在 Linux 上以[All-in-one](https://kubesphere.io/zh/docs/quick-start/all-in-one-on-linux/) 模式来安装单节点 Kubernetes 和 KubeSphere。
+
+### 在托管 Kubernetes 上部署 KubeSphere
+
+KubeSphere 托管在以下云供应商上，您可以通过在其托管的 Kubernetes 服务上一键安装来部署 KubeSphere。
+
+- [在 Amazon EKS 上部署 KubeSphere](https://aws.amazon.com/quickstart/architecture/qingcloud-kubesphere/)
+- [在 Azure AKS 上部署 KubeSphere](https://market.azure.cn/marketplace/apps/qingcloud.kubesphere)
+- [在 DigitalOcean 上部署 KubeSphere](https://marketplace.digitalocean.com/apps/kubesphere)
+- [在青云QingCloud QKE 上部署 KubeSphere](https://www.qingcloud.com/products/kubesphereqke)
+
+您还可以在几分钟内在其他托管的 Kubernetes 服务上安装 KubeSphere，请参阅 [官方文档](https://kubesphere.io/zh/docs/installing-on-kubernetes/) 以开始使用。
+
+> 👨‍💻 不能访问网络？参考 [在Kubernetes上离线安装](https://kubesphere.io/zh/docs/installing-on-kubernetes/on-prem-kubernetes/install-ks-on-linux-airgapped/) 或者 [在 Linux 上离线安装](https://kubesphere.io/zh/docs/installing-on-linux/introduction/air-gapped-installation/) 了解如何使用私有仓库来安装 KubeSphere。
+
+## 贡献、支持、讨论和社区
+
+我们 :heart: 您的贡献。[社区](https://github.com/kubesphere/community) 将引导您了解如何开始贡献 KubeSphere。[开发指南](https://github.com/kubesphere/community/tree/master/developer-guide/development) 说明了如何安装开发环境。
+
+- [中文论坛](https://kubesphere.com.cn/forum/)
+- [社区微信群（见官网底部）](https://kubesphere.com.cn/)
+- [Slack Channel](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE)
+- [Youtube](https://www.youtube.com/channel/UCyTdUQUYjf7XLjxECx63Hpw)
+- [在推特上关注我们](https://twitter.com/KubeSphere)
+
+请将任何 KubeSphere 错误、问题和功能请求提交到 [KubeSphere GitHub Issue](https://github.com/kubesphere/kubesphere/issues) 。
+
+## 谁在使用 KubeSphere
+
+[用户案例学习](https://kubesphere.com.cn/case/) 列出了哪些企业在使用 KubeSphere。欢迎 [发表评论](https://github.com/kubesphere/kubesphere/issues/4123) 来分享您的使用案例。
+
+## Landscapes
+
+<p align="center">
+<br/><br/>
+<img src="https://landscape.cncf.io/images/left-logo.svg" width="150"/>&nbsp;&nbsp;<img src="https://landscape.cncf.io/images/right-logo.svg" width="200"/>&nbsp;&nbsp;
+<br/><br/>
+KubeSphere 是 CNCF 基金会成员并且通过了 <a href="https://www.cncf.io/certification/software-conformance/#logos">Kubernetes 一致性认证
+</a>，进一步丰富了 <a href="https://landscape.cncf.io/?landscape=observability-and-analysis&license=apache-license-2-0">CNCF 云原生的生态。
+</a>
+</p>

api/api-rules/violation_exceptions.list

@@ -0,0 +1,95 @@
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIGroup,ServerAddressByClientCIDRs
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIGroup,Versions
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIGroupList,Groups
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIResource,Categories
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIResource,ShortNames
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIResourceList,APIResources
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIVersions,ServerAddressByClientCIDRs
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIVersions,Versions
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,ApplyOptions,DryRun
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,CreateOptions,DryRun
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,DeleteOptions,DryRun
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,FieldsV1,Raw
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,LabelSelector,MatchExpressions
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,LabelSelectorRequirement,Values
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,ObjectMeta,Finalizers
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,ObjectMeta,ManagedFields
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,ObjectMeta,OwnerReferences
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,PatchOptions,DryRun
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,RootPaths,Paths
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,StatusDetails,Causes
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,Table,ColumnDefinitions
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,Table,Rows
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,TableRow,Cells
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,TableRow,Conditions
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,UpdateOptions,DryRun
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/runtime,RawExtension,Raw
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/runtime,Unknown,Raw
+API rule violation: list_type_missing,kubesphere.io/api/devops/v1alpha3,NoScmPipeline,Parameters
+API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,APIResourceList,APIResources
+API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,Duration,Duration
+API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,InternalEvent,Object
+API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,InternalEvent,Type
+API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,MicroTime,Time
+API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,StatusCause,Type
+API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,Time,Time
+API rule violation: names_match,k8s.io/apimachinery/pkg/runtime,Unknown,ContentEncoding
+API rule violation: names_match,k8s.io/apimachinery/pkg/runtime,Unknown,ContentType
+API rule violation: names_match,k8s.io/apimachinery/pkg/runtime,Unknown,Raw
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,BitbucketServerSource,ApiUri
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,BitbucketServerSource,CloneOption
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,BitbucketServerSource,CredentialId
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,BitbucketServerSource,DiscoverBranches
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,BitbucketServerSource,DiscoverPRFromForks
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,BitbucketServerSource,DiscoverPRFromOrigin
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,BitbucketServerSource,DiscoverTags
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,BitbucketServerSource,RegexFilter
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,BitbucketServerSource,ScmId
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,DiscarderProperty,DaysToKeep
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,DiscarderProperty,NumToKeep
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitSource,CloneOption
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitSource,CredentialId
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitSource,DiscoverBranches
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitSource,DiscoverTags
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitSource,RegexFilter
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitSource,ScmId
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GithubSource,ApiUri
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GithubSource,CloneOption
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GithubSource,CredentialId
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GithubSource,DiscoverBranches
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GithubSource,DiscoverPRFromForks
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GithubSource,DiscoverPRFromOrigin
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GithubSource,DiscoverTags
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GithubSource,RegexFilter
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GithubSource,ScmId
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitlabSource,ApiUri
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitlabSource,CloneOption
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitlabSource,CredentialId
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitlabSource,DiscoverBranches
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitlabSource,DiscoverPRFromForks
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitlabSource,DiscoverPRFromOrigin
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitlabSource,DiscoverTags
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitlabSource,RegexFilter
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitlabSource,ScmId
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,GitlabSource,ServerName
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchJobTrigger,CreateActionJobsToTrigger
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchJobTrigger,DeleteActionJobsToTrigger
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchPipeline,BitbucketServerSource
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchPipeline,GitHubSource
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchPipeline,GitSource
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchPipeline,GitlabSource
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchPipeline,MultiBranchJobTrigger
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchPipeline,ScriptPath
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchPipeline,SingleSvnSource
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchPipeline,SourceType
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchPipeline,SvnSource
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,MultiBranchPipeline,TimerTrigger
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,NoScmPipeline,DisableConcurrent
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,NoScmPipeline,RemoteTrigger
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,NoScmPipeline,TimerTrigger
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,Parameter,DefaultValue
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,PipelineSpec,MultiBranchPipeline
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,SingleSvnSource,CredentialId
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,SingleSvnSource,ScmId
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,SvnSource,CredentialId
+API rule violation: names_match,kubesphere.io/api/devops/v1alpha3,SvnSource,ScmId

build/builder-docker/.ash_history

@@ -1,3 +0,0 @@
-ls /go/bin
-go version
-exit

build/builder-docker/Dockerfile

@@ -1,19 +0,0 @@
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
-# that can be found in the LICENSE file.
-
-FROM golang:1.10.2-alpine3.7 as builder
-
-RUN apk add --no-cache git curl openssl
-
-RUN go get github.com/tools/godep
-#RUN go get github.com/emicklei/go-restful
-#RUN go get github.com/golang/glog
-#RUN go get github.com/spf13/pflag
-RUN go get golang.org/x/tools/cmd/goimports
-
-FROM golang:1.10.2-alpine3.7
-
-RUN apk add --no-cache git make curl openssl jq rsync godep
-
-COPY --from=builder /go/bin /go/bin

build/builder-docker/Makefile

@@ -1,17 +0,0 @@
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
-# that can be found in the LICENSE file.
-
-default:
-	docker build -t kubesphere/kubesphere-builder .
-	@echo "ok"
-
-pull:
-	docker pull kubesphere/kubesphere-builder
-	@echo "ok"
-
-run:
-	docker run --rm -it -v `pwd`:/root kubesphere/kubesphere-builder
-
-clean:
-	@echo "ok"

build/ks-apiserver/Dockerfile

@@ -0,0 +1,48 @@
+# Copyright 2020 The KubeSphere Authors. All rights reserved.
+# Use of this source code is governed by an Apache license
+# that can be found in the LICENSE file.
+
+# Download dependencies
+FROM alpine:3.11 as base_os_context
+
+ARG TARGETARCH
+ARG TARGETOS
+ARG HELM_VERSION=v3.5.2
+
+ENV OUTDIR=/out
+RUN mkdir -p ${OUTDIR}/usr/local/bin/
+
+WORKDIR /tmp
+
+RUN apk add --no-cache ca-certificates
+
+# install helm
+ADD https://get.helm.sh/helm-${HELM_VERSION}-${TARGETOS}-${TARGETARCH}.tar.gz /tmp
+RUN tar xvzf /tmp/helm-${HELM_VERSION}-${TARGETOS}-${TARGETARCH}.tar.gz -C /tmp
+RUN mv /tmp/${TARGETOS}-${TARGETARCH}/helm ${OUTDIR}/usr/local/bin/
+
+# Build 
+FROM golang:1.16.3 as build_context
+
+ENV OUTDIR=/out
+RUN mkdir -p ${OUTDIR}/usr/local/bin/
+
+WORKDIR /workspace
+ADD . /workspace/
+
+RUN make ks-apiserver
+RUN mv /workspace/bin/cmd/ks-apiserver ${OUTDIR}/usr/local/bin/
+
+##############
+# Final image
+#############
+
+FROM alpine:3.11 
+
+COPY --from=base_os_context /out/ /
+COPY --from=build_context /out/ /
+
+WORKDIR /
+
+EXPOSE 9090
+CMD ["sh"]

build/ks-controller-manager/Dockerfile

@@ -0,0 +1,67 @@
+# Copyright 2020 The KubeSphere Authors. All rights reserved.
+# Use of this source code is governed by an Apache license
+# that can be found in the LICENSE file.
+
+# Download dependencies
+FROM alpine:3.11 as base_os_context
+
+ARG TARGETARCH
+ARG TARGETOS
+ARG HELM_VERSION=v3.5.2
+ARG KUSTOMIZE_VERSION=v4.2.0
+ARG INGRESS_NGINX_VERSION=3.35.0
+
+ENV OUTDIR=/out
+RUN mkdir -p ${OUTDIR}/usr/local/bin
+RUN mkdir -p ${OUTDIR}/var/helm-charts
+
+WORKDIR /tmp
+
+RUN apk add --no-cache ca-certificates
+
+# Install helm
+ADD https://get.helm.sh/helm-${HELM_VERSION}-${TARGETOS}-${TARGETARCH}.tar.gz /tmp
+RUN tar xvzf /tmp/helm-${HELM_VERSION}-${TARGETOS}-${TARGETARCH}.tar.gz -C /tmp
+RUN mv /tmp/${TARGETOS}-${TARGETARCH}/helm ${OUTDIR}/usr/local/bin/
+
+# install kustomize
+ADD https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${KUSTOMIZE_VERSION}/kustomize_${KUSTOMIZE_VERSION}_${TARGETOS}_${TARGETARCH}.tar.gz /tmp
+RUN tar xvzf /tmp/kustomize_${KUSTOMIZE_VERSION}_linux_${TARGETARCH}.tar.gz -C /tmp
+RUN mv /tmp/kustomize ${OUTDIR}/usr/local/bin/
+
+
+# Install Nginx Ingress Helm Chart
+ADD https://github.com/kubernetes/ingress-nginx/releases/download/helm-chart-${INGRESS_NGINX_VERSION}/ingress-nginx-${INGRESS_NGINX_VERSION}.tgz /tmp
+RUN tar xvzf /tmp/ingress-nginx-${INGRESS_NGINX_VERSION}.tgz -C /tmp
+RUN mv /tmp/ingress-nginx ${OUTDIR}/var/helm-charts/
+
+# Build
+
+FROM golang:1.16.3 as build_context
+
+ENV OUTDIR=/out
+RUN mkdir -p ${OUTDIR}/usr/local/bin/
+RUN mkdir -p ${OUTDIR}/var/helm-charts
+
+WORKDIR /workspace
+ADD . /workspace/
+
+RUN make ks-controller-manager
+RUN mv /workspace/bin/cmd/controller-manager ${OUTDIR}/usr/local/bin/
+
+# Copy gateway config and helm chart
+RUN mv /workspace/config/gateway ${OUTDIR}/var/helm-charts/
+RUN mv /workspace/config/watches.yaml ${OUTDIR}/var/helm-charts/
+
+# Final Image
+
+FROM alpine:3.11 
+
+COPY --from=base_os_context /out/ /
+COPY --from=build_context /out/ /
+
+WORKDIR /
+
+EXPOSE 8443 8080
+
+CMD ["sh"]

cmd/controller-manager/app/controllers.go

@@ -0,0 +1,243 @@
+/*
+Copyright 2019 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package app
+
+import (
+	"k8s.io/client-go/tools/cache"
+	"k8s.io/klog"
+	runtimeclient "sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/manager"
+	"sigs.k8s.io/kubefed/pkg/controller/util"
+
+	"kubesphere.io/kubesphere/pkg/controller/storage/snapshotclass"
+
+	"kubesphere.io/kubesphere/pkg/apiserver/authentication"
+
+	iamv1alpha2 "kubesphere.io/api/iam/v1alpha2"
+
+	"kubesphere.io/kubesphere/pkg/controller/certificatesigningrequest"
+	"kubesphere.io/kubesphere/pkg/controller/cluster"
+	"kubesphere.io/kubesphere/pkg/controller/clusterrolebinding"
+	"kubesphere.io/kubesphere/pkg/controller/destinationrule"
+	"kubesphere.io/kubesphere/pkg/controller/globalrole"
+	"kubesphere.io/kubesphere/pkg/controller/globalrolebinding"
+	"kubesphere.io/kubesphere/pkg/controller/group"
+	"kubesphere.io/kubesphere/pkg/controller/groupbinding"
+	"kubesphere.io/kubesphere/pkg/controller/job"
+	"kubesphere.io/kubesphere/pkg/controller/loginrecord"
+	"kubesphere.io/kubesphere/pkg/controller/network/ippool"
+	"kubesphere.io/kubesphere/pkg/controller/network/nsnetworkpolicy"
+	"kubesphere.io/kubesphere/pkg/controller/network/nsnetworkpolicy/provider"
+	"kubesphere.io/kubesphere/pkg/controller/notification"
+	"kubesphere.io/kubesphere/pkg/controller/storage/capability"
+	"kubesphere.io/kubesphere/pkg/controller/virtualservice"
+	"kubesphere.io/kubesphere/pkg/informers"
+	"kubesphere.io/kubesphere/pkg/simple/client/devops"
+	"kubesphere.io/kubesphere/pkg/simple/client/k8s"
+	ldapclient "kubesphere.io/kubesphere/pkg/simple/client/ldap"
+	"kubesphere.io/kubesphere/pkg/simple/client/multicluster"
+	"kubesphere.io/kubesphere/pkg/simple/client/network"
+	ippoolclient "kubesphere.io/kubesphere/pkg/simple/client/network/ippool"
+	"kubesphere.io/kubesphere/pkg/simple/client/s3"
+)
+
+func addControllers(
+	mgr manager.Manager,
+	client k8s.Client,
+	informerFactory informers.InformerFactory,
+	devopsClient devops.Interface,
+	s3Client s3.Interface,
+	ldapClient ldapclient.Interface,
+	options *k8s.KubernetesOptions,
+	authenticationOptions *authentication.Options,
+	multiClusterOptions *multicluster.Options,
+	networkOptions *network.Options,
+	serviceMeshEnabled bool,
+	kubectlImage string,
+	stopCh <-chan struct{}) error {
+
+	kubernetesInformer := informerFactory.KubernetesSharedInformerFactory()
+	istioInformer := informerFactory.IstioSharedInformerFactory()
+	kubesphereInformer := informerFactory.KubeSphereSharedInformerFactory()
+
+	multiClusterEnabled := multiClusterOptions.Enable
+
+	var vsController, drController manager.Runnable
+	if serviceMeshEnabled {
+		vsController = virtualservice.NewVirtualServiceController(kubernetesInformer.Core().V1().Services(),
+			istioInformer.Networking().V1alpha3().VirtualServices(),
+			istioInformer.Networking().V1alpha3().DestinationRules(),
+			kubesphereInformer.Servicemesh().V1alpha2().Strategies(),
+			client.Kubernetes(),
+			client.Istio(),
+			client.KubeSphere())
+
+		drController = destinationrule.NewDestinationRuleController(kubernetesInformer.Apps().V1().Deployments(),
+			istioInformer.Networking().V1alpha3().DestinationRules(),
+			kubernetesInformer.Core().V1().Services(),
+			kubesphereInformer.Servicemesh().V1alpha2().ServicePolicies(),
+			client.Kubernetes(),
+			client.Istio(),
+			client.KubeSphere())
+	}
+
+	jobController := job.NewJobController(kubernetesInformer.Batch().V1().Jobs(), client.Kubernetes())
+
+	storageCapabilityController := capability.NewController(
+		client.Kubernetes().StorageV1().StorageClasses(),
+		kubernetesInformer.Storage().V1().StorageClasses(),
+		kubernetesInformer.Storage().V1().CSIDrivers(),
+	)
+
+	volumeSnapshotController := snapshotclass.NewController(
+		kubernetesInformer.Storage().V1().StorageClasses(),
+		client.Snapshot().SnapshotV1().VolumeSnapshotClasses(),
+		informerFactory.SnapshotSharedInformerFactory().Snapshot().V1().VolumeSnapshotClasses(),
+	)
+
+	var fedGlobalRoleBindingCache, fedGlobalRoleCache cache.Store
+	var fedGlobalRoleBindingCacheController, fedGlobalRoleCacheController cache.Controller
+
+	if multiClusterEnabled {
+		fedGlobalRoleClient, err := util.NewResourceClient(client.Config(), &iamv1alpha2.FedGlobalRoleResource)
+		if err != nil {
+			klog.Error(err)
+			return err
+		}
+		fedGlobalRoleBindingClient, err := util.NewResourceClient(client.Config(), &iamv1alpha2.FedGlobalRoleBindingResource)
+		if err != nil {
+			klog.Error(err)
+			return err
+		}
+
+		fedGlobalRoleCache, fedGlobalRoleCacheController = util.NewResourceInformer(fedGlobalRoleClient, "", &iamv1alpha2.FedGlobalRoleResource, func(object runtimeclient.Object) {})
+		fedGlobalRoleBindingCache, fedGlobalRoleBindingCacheController = util.NewResourceInformer(fedGlobalRoleBindingClient, "", &iamv1alpha2.FedGlobalRoleBindingResource, func(object runtimeclient.Object) {})
+
+		go fedGlobalRoleCacheController.Run(stopCh)
+		go fedGlobalRoleBindingCacheController.Run(stopCh)
+	}
+
+	loginRecordController := loginrecord.NewLoginRecordController(
+		client.Kubernetes(),
+		client.KubeSphere(),
+		kubesphereInformer.Iam().V1alpha2().LoginRecords(),
+		kubesphereInformer.Iam().V1alpha2().Users(),
+		authenticationOptions.LoginHistoryRetentionPeriod,
+		authenticationOptions.LoginHistoryMaximumEntries)
+
+	csrController := certificatesigningrequest.NewController(client.Kubernetes(),
+		kubernetesInformer.Certificates().V1().CertificateSigningRequests(),
+		kubernetesInformer.Core().V1().ConfigMaps(), client.Config())
+
+	clusterRoleBindingController := clusterrolebinding.NewController(client.Kubernetes(),
+		kubernetesInformer.Rbac().V1().ClusterRoleBindings(),
+		kubernetesInformer.Apps().V1().Deployments(),
+		kubernetesInformer.Core().V1().Pods(),
+		kubesphereInformer.Iam().V1alpha2().Users(),
+		kubectlImage)
+
+	globalRoleController := globalrole.NewController(client.Kubernetes(), client.KubeSphere(),
+		kubesphereInformer.Iam().V1alpha2().GlobalRoles(), fedGlobalRoleCache, fedGlobalRoleCacheController)
+
+	globalRoleBindingController := globalrolebinding.NewController(client.Kubernetes(), client.KubeSphere(),
+		kubesphereInformer.Iam().V1alpha2().GlobalRoleBindings(),
+		fedGlobalRoleBindingCache, fedGlobalRoleBindingCacheController,
+		multiClusterEnabled)
+
+	groupBindingController := groupbinding.NewController(client.Kubernetes(), client.KubeSphere(),
+		kubesphereInformer.Iam().V1alpha2().GroupBindings(),
+		kubesphereInformer.Types().V1beta1().FederatedGroupBindings(),
+		multiClusterEnabled)
+
+	groupController := group.NewController(client.Kubernetes(), client.KubeSphere(),
+		kubesphereInformer.Iam().V1alpha2().Groups(),
+		kubesphereInformer.Types().V1beta1().FederatedGroups(),
+		multiClusterEnabled)
+
+	var clusterController manager.Runnable
+	if multiClusterEnabled {
+		clusterController = cluster.NewClusterController(
+			client.Kubernetes(),
+			client.Config(),
+			kubesphereInformer.Cluster().V1alpha1().Clusters(),
+			client.KubeSphere().ClusterV1alpha1().Clusters(),
+			multiClusterOptions.ClusterControllerResyncPeriod,
+			multiClusterOptions.HostClusterName)
+	}
+
+	var nsnpController manager.Runnable
+	if networkOptions.EnableNetworkPolicy {
+		nsnpProvider, err := provider.NewNsNetworkPolicyProvider(client.Kubernetes(), kubernetesInformer.Networking().V1().NetworkPolicies())
+		if err != nil {
+			return err
+		}
+
+		nsnpController = nsnetworkpolicy.NewNSNetworkPolicyController(client.Kubernetes(),
+			client.KubeSphere().NetworkV1alpha1(),
+			kubesphereInformer.Network().V1alpha1().NamespaceNetworkPolicies(),
+			kubernetesInformer.Core().V1().Services(),
+			kubernetesInformer.Core().V1().Nodes(),
+			kubesphereInformer.Tenant().V1alpha1().Workspaces(),
+			kubernetesInformer.Core().V1().Namespaces(), nsnpProvider, networkOptions.NSNPOptions)
+	}
+
+	var ippoolController manager.Runnable
+	ippoolProvider := ippoolclient.NewProvider(kubernetesInformer, client.KubeSphere(), client.Kubernetes(), networkOptions.IPPoolType, options)
+	if ippoolProvider != nil {
+		ippoolController = ippool.NewIPPoolController(kubesphereInformer, kubernetesInformer, client.Kubernetes(), client.KubeSphere(), ippoolProvider)
+	}
+
+	controllers := map[string]manager.Runnable{
+		"virtualservice-controller":     vsController,
+		"destinationrule-controller":    drController,
+		"job-controller":                jobController,
+		"storagecapability-controller":  storageCapabilityController,
+		"volumesnapshot-controller":     volumeSnapshotController,
+		"loginrecord-controller":        loginRecordController,
+		"cluster-controller":            clusterController,
+		"nsnp-controller":               nsnpController,
+		"csr-controller":                csrController,
+		"clusterrolebinding-controller": clusterRoleBindingController,
+		"globalrolebinding-controller":  globalRoleBindingController,
+		"ippool-controller":             ippoolController,
+		"groupbinding-controller":       groupBindingController,
+		"group-controller":              groupController,
+	}
+
+	if multiClusterEnabled {
+		controllers["globalrole-controller"] = globalRoleController
+		notificationController, err := notification.NewController(client.Kubernetes(), mgr.GetClient(), mgr.GetCache())
+		if err != nil {
+			return err
+		}
+		controllers["notification-controller"] = notificationController
+	}
+
+	for name, ctrl := range controllers {
+		if ctrl == nil {
+			klog.V(4).Infof("%s is not going to run due to dependent component disabled.", name)
+			continue
+		}
+
+		if err := mgr.Add(ctrl); err != nil {
+			klog.Error(err, "add controller to manager failed", "name", name)
+			return err
+		}
+	}
+
+	return nil
+}

cmd/controller-manager/app/helper.go

@@ -0,0 +1,56 @@
+/*
+Copyright 2020 KubeSphere Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package app
+
+import (
+	"context"
+	"fmt"
+	"net/http"
+	"time"
+
+	"k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
+	"k8s.io/apimachinery/pkg/util/wait"
+	"k8s.io/klog"
+)
+
+// WaitForAPIServer waits for the API Server's /healthz endpoint to report "ok" before timeout.
+func WaitForAPIServer(client clientset.Interface, timeout time.Duration) error {
+	var lastErr error
+
+	err := wait.PollImmediate(time.Second, timeout, func() (bool, error) {
+		healthStatus := 0
+		result := client.Discovery().RESTClient().Get().AbsPath("/healthz").Do(context.Background()).StatusCode(&healthStatus)
+		if result.Error() != nil {
+			lastErr = fmt.Errorf("failed to get apiserver /healthz status: %v", result.Error())
+			return false, nil
+		}
+		if healthStatus != http.StatusOK {
+			content, _ := result.Raw()
+			lastErr = fmt.Errorf("APIServer isn't healthy: %v", string(content))
+			klog.Warningf("APIServer isn't healthy yet: %v. Waiting a little while.", string(content))
+			return false, nil
+		}
+
+		return true, nil
+	})
+
+	if err != nil {
+		return fmt.Errorf("%v: %v", err, lastErr)
+	}
+
+	return nil
+}

cmd/controller-manager/app/options/options.go

@@ -0,0 +1,169 @@
+/*
+Copyright 2020 KubeSphere Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package options
+
+import (
+	"flag"
+	"strings"
+	"time"
+
+	"kubesphere.io/kubesphere/pkg/apiserver/authentication"
+
+	"k8s.io/apimachinery/pkg/labels"
+
+	"github.com/spf13/pflag"
+	"k8s.io/client-go/tools/leaderelection"
+	cliflag "k8s.io/component-base/cli/flag"
+	"k8s.io/klog"
+
+	"kubesphere.io/kubesphere/pkg/simple/client/devops/jenkins"
+	"kubesphere.io/kubesphere/pkg/simple/client/gateway"
+	"kubesphere.io/kubesphere/pkg/simple/client/k8s"
+	ldapclient "kubesphere.io/kubesphere/pkg/simple/client/ldap"
+	"kubesphere.io/kubesphere/pkg/simple/client/multicluster"
+	"kubesphere.io/kubesphere/pkg/simple/client/network"
+	"kubesphere.io/kubesphere/pkg/simple/client/openpitrix"
+	"kubesphere.io/kubesphere/pkg/simple/client/s3"
+	"kubesphere.io/kubesphere/pkg/simple/client/servicemesh"
+)
+
+type KubeSphereControllerManagerOptions struct {
+	KubernetesOptions     *k8s.KubernetesOptions
+	DevopsOptions         *jenkins.Options
+	S3Options             *s3.Options
+	AuthenticationOptions *authentication.Options
+	LdapOptions           *ldapclient.Options
+	OpenPitrixOptions     *openpitrix.Options
+	NetworkOptions        *network.Options
+	MultiClusterOptions   *multicluster.Options
+	ServiceMeshOptions    *servicemesh.Options
+	GatewayOptions        *gateway.Options
+	LeaderElect           bool
+	LeaderElection        *leaderelection.LeaderElectionConfig
+	WebhookCertDir        string
+
+	// KubeSphere is using sigs.k8s.io/application as fundamental object to implement Application Management.
+	// There are other projects also built on sigs.k8s.io/application, when KubeSphere installed along side
+	// them, conflicts happen. So we leave an option to only reconcile applications  matched with the given
+	// selector. Default will reconcile all applications.
+	//    For example
+	//      "kubesphere.io/creator=" means reconcile applications with this label key
+	//      "!kubesphere.io/creator" means exclude applications with this key
+	ApplicationSelector string
+}
+
+func NewKubeSphereControllerManagerOptions() *KubeSphereControllerManagerOptions {
+	s := &KubeSphereControllerManagerOptions{
+		KubernetesOptions:     k8s.NewKubernetesOptions(),
+		DevopsOptions:         jenkins.NewDevopsOptions(),
+		S3Options:             s3.NewS3Options(),
+		LdapOptions:           ldapclient.NewOptions(),
+		OpenPitrixOptions:     openpitrix.NewOptions(),
+		NetworkOptions:        network.NewNetworkOptions(),
+		MultiClusterOptions:   multicluster.NewOptions(),
+		ServiceMeshOptions:    servicemesh.NewServiceMeshOptions(),
+		AuthenticationOptions: authentication.NewOptions(),
+		GatewayOptions:        gateway.NewGatewayOptions(),
+		LeaderElection: &leaderelection.LeaderElectionConfig{
+			LeaseDuration: 30 * time.Second,
+			RenewDeadline: 15 * time.Second,
+			RetryPeriod:   5 * time.Second,
+		},
+		LeaderElect:         false,
+		WebhookCertDir:      "",
+		ApplicationSelector: "",
+	}
+
+	return s
+}
+
+func (s *KubeSphereControllerManagerOptions) Flags() cliflag.NamedFlagSets {
+	fss := cliflag.NamedFlagSets{}
+
+	s.KubernetesOptions.AddFlags(fss.FlagSet("kubernetes"), s.KubernetesOptions)
+	s.DevopsOptions.AddFlags(fss.FlagSet("devops"), s.DevopsOptions)
+	s.S3Options.AddFlags(fss.FlagSet("s3"), s.S3Options)
+	s.AuthenticationOptions.AddFlags(fss.FlagSet("authentication"), s.AuthenticationOptions)
+	s.LdapOptions.AddFlags(fss.FlagSet("ldap"), s.LdapOptions)
+	s.OpenPitrixOptions.AddFlags(fss.FlagSet("openpitrix"), s.OpenPitrixOptions)
+	s.NetworkOptions.AddFlags(fss.FlagSet("network"), s.NetworkOptions)
+	s.MultiClusterOptions.AddFlags(fss.FlagSet("multicluster"), s.MultiClusterOptions)
+	s.ServiceMeshOptions.AddFlags(fss.FlagSet("servicemesh"), s.ServiceMeshOptions)
+	s.GatewayOptions.AddFlags(fss.FlagSet("gateway"), s.GatewayOptions)
+	fs := fss.FlagSet("leaderelection")
+	s.bindLeaderElectionFlags(s.LeaderElection, fs)
+
+	fs.BoolVar(&s.LeaderElect, "leader-elect", s.LeaderElect, ""+
+		"Whether to enable leader election. This field should be enabled when controller manager"+
+		"deployed with multiple replicas.")
+
+	fs.StringVar(&s.WebhookCertDir, "webhook-cert-dir", s.WebhookCertDir, ""+
+		"Certificate directory used to setup webhooks, need tls.crt and tls.key placed inside."+
+		"if not set, webhook server would look up the server key and certificate in"+
+		"{TempDir}/k8s-webhook-server/serving-certs")
+
+	gfs := fss.FlagSet("generic")
+	gfs.StringVar(&s.ApplicationSelector, "application-selector", s.ApplicationSelector, ""+
+		"Only reconcile application(sigs.k8s.io/application) objects match given selector, this could avoid conflicts with "+
+		"other projects built on top of sig-application. Default behavior is to reconcile all of application objects.")
+
+	kfs := fss.FlagSet("klog")
+	local := flag.NewFlagSet("klog", flag.ExitOnError)
+	klog.InitFlags(local)
+	local.VisitAll(func(fl *flag.Flag) {
+		fl.Name = strings.Replace(fl.Name, "_", "-", -1)
+		kfs.AddGoFlag(fl)
+	})
+
+	return fss
+}
+
+func (s *KubeSphereControllerManagerOptions) Validate() []error {
+	var errs []error
+	errs = append(errs, s.DevopsOptions.Validate()...)
+	errs = append(errs, s.KubernetesOptions.Validate()...)
+	errs = append(errs, s.S3Options.Validate()...)
+	errs = append(errs, s.OpenPitrixOptions.Validate()...)
+	errs = append(errs, s.NetworkOptions.Validate()...)
+	errs = append(errs, s.LdapOptions.Validate()...)
+	errs = append(errs, s.MultiClusterOptions.Validate()...)
+
+	if len(s.ApplicationSelector) != 0 {
+		_, err := labels.Parse(s.ApplicationSelector)
+		if err != nil {
+			errs = append(errs, err)
+		}
+	}
+
+	return errs
+}
+
+func (s *KubeSphereControllerManagerOptions) bindLeaderElectionFlags(l *leaderelection.LeaderElectionConfig, fs *pflag.FlagSet) {
+	fs.DurationVar(&l.LeaseDuration, "leader-elect-lease-duration", l.LeaseDuration, ""+
+		"The duration that non-leader candidates will wait after observing a leadership "+
+		"renewal until attempting to acquire leadership of a led but unrenewed leader "+
+		"slot. This is effectively the maximum duration that a leader can be stopped "+
+		"before it is replaced by another candidate. This is only applicable if leader "+
+		"election is enabled.")
+	fs.DurationVar(&l.RenewDeadline, "leader-elect-renew-deadline", l.RenewDeadline, ""+
+		"The interval between attempts by the acting master to renew a leadership slot "+
+		"before it stops leading. This must be less than or equal to the lease duration. "+
+		"This is only applicable if leader election is enabled.")
+	fs.DurationVar(&l.RetryPeriod, "leader-elect-retry-period", l.RetryPeriod, ""+
+		"The duration the clients should wait between attempting acquisition and renewal "+
+		"of a leadership. This is only applicable if leader election is enabled.")
+}

cmd/controller-manager/app/server.go

@@ -0,0 +1,372 @@
+/*
+Copyright 2019 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package app
+
+import (
+	"context"
+	"fmt"
+	"os"
+
+	"kubesphere.io/kubesphere/pkg/models/kubeconfig"
+
+	"github.com/spf13/cobra"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
+	utilerrors "k8s.io/apimachinery/pkg/util/errors"
+	cliflag "k8s.io/component-base/cli/flag"
+	"k8s.io/klog"
+	"k8s.io/klog/klogr"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/manager"
+	"sigs.k8s.io/controller-runtime/pkg/manager/signals"
+	"sigs.k8s.io/controller-runtime/pkg/webhook"
+
+	"kubesphere.io/kubesphere/cmd/controller-manager/app/options"
+	"kubesphere.io/kubesphere/pkg/apis"
+	controllerconfig "kubesphere.io/kubesphere/pkg/apiserver/config"
+	"kubesphere.io/kubesphere/pkg/controller/application"
+	"kubesphere.io/kubesphere/pkg/controller/helm"
+	"kubesphere.io/kubesphere/pkg/controller/namespace"
+	"kubesphere.io/kubesphere/pkg/controller/network/webhooks"
+	"kubesphere.io/kubesphere/pkg/controller/openpitrix/helmapplication"
+	"kubesphere.io/kubesphere/pkg/controller/openpitrix/helmcategory"
+	"kubesphere.io/kubesphere/pkg/controller/openpitrix/helmrelease"
+	"kubesphere.io/kubesphere/pkg/controller/openpitrix/helmrepo"
+	"kubesphere.io/kubesphere/pkg/controller/quota"
+	"kubesphere.io/kubesphere/pkg/controller/serviceaccount"
+	"kubesphere.io/kubesphere/pkg/controller/user"
+	"kubesphere.io/kubesphere/pkg/controller/workspace"
+	"kubesphere.io/kubesphere/pkg/controller/workspacerole"
+	"kubesphere.io/kubesphere/pkg/controller/workspacerolebinding"
+	"kubesphere.io/kubesphere/pkg/controller/workspacetemplate"
+	"kubesphere.io/kubesphere/pkg/informers"
+	"kubesphere.io/kubesphere/pkg/simple/client/devops"
+	"kubesphere.io/kubesphere/pkg/simple/client/devops/jenkins"
+	"kubesphere.io/kubesphere/pkg/simple/client/k8s"
+	ldapclient "kubesphere.io/kubesphere/pkg/simple/client/ldap"
+	"kubesphere.io/kubesphere/pkg/simple/client/s3"
+	"kubesphere.io/kubesphere/pkg/utils/metrics"
+	"kubesphere.io/kubesphere/pkg/utils/term"
+	"kubesphere.io/kubesphere/pkg/version"
+)
+
+func NewControllerManagerCommand() *cobra.Command {
+	s := options.NewKubeSphereControllerManagerOptions()
+	conf, err := controllerconfig.TryLoadFromDisk()
+	if err == nil {
+		// make sure LeaderElection is not nil
+		s = &options.KubeSphereControllerManagerOptions{
+			KubernetesOptions:     conf.KubernetesOptions,
+			DevopsOptions:         conf.DevopsOptions,
+			S3Options:             conf.S3Options,
+			AuthenticationOptions: conf.AuthenticationOptions,
+			LdapOptions:           conf.LdapOptions,
+			OpenPitrixOptions:     conf.OpenPitrixOptions,
+			NetworkOptions:        conf.NetworkOptions,
+			MultiClusterOptions:   conf.MultiClusterOptions,
+			ServiceMeshOptions:    conf.ServiceMeshOptions,
+			GatewayOptions:        conf.GatewayOptions,
+			LeaderElection:        s.LeaderElection,
+			LeaderElect:           s.LeaderElect,
+			WebhookCertDir:        s.WebhookCertDir,
+		}
+	} else {
+		klog.Fatal("Failed to load configuration from disk", err)
+	}
+
+	cmd := &cobra.Command{
+		Use:  "controller-manager",
+		Long: `KubeSphere controller manager is a daemon that`,
+		Run: func(cmd *cobra.Command, args []string) {
+			if errs := s.Validate(); len(errs) != 0 {
+				klog.Error(utilerrors.NewAggregate(errs))
+				os.Exit(1)
+			}
+
+			if err = run(s, signals.SetupSignalHandler()); err != nil {
+				klog.Error(err)
+				os.Exit(1)
+			}
+		},
+		SilenceUsage: true,
+	}
+
+	fs := cmd.Flags()
+	namedFlagSets := s.Flags()
+
+	for _, f := range namedFlagSets.FlagSets {
+		fs.AddFlagSet(f)
+	}
+
+	usageFmt := "Usage:\n  %s\n"
+	cols, _, _ := term.TerminalSize(cmd.OutOrStdout())
+	cmd.SetHelpFunc(func(cmd *cobra.Command, args []string) {
+		_, _ = fmt.Fprintf(cmd.OutOrStdout(), "%s\n\n"+usageFmt, cmd.Long, cmd.UseLine())
+		cliflag.PrintSections(cmd.OutOrStdout(), namedFlagSets, cols)
+	})
+
+	versionCmd := &cobra.Command{
+		Use:   "version",
+		Short: "Print the version of KubeSphere controller-manager",
+		Run: func(cmd *cobra.Command, args []string) {
+			cmd.Println(version.Get())
+		},
+	}
+
+	cmd.AddCommand(versionCmd)
+
+	return cmd
+}
+
+func run(s *options.KubeSphereControllerManagerOptions, ctx context.Context) error {
+
+	kubernetesClient, err := k8s.NewKubernetesClient(s.KubernetesOptions)
+	if err != nil {
+		klog.Errorf("Failed to create kubernetes clientset %v", err)
+		return err
+	}
+
+	var devopsClient devops.Interface
+	if s.DevopsOptions != nil && len(s.DevopsOptions.Host) != 0 {
+		devopsClient, err = jenkins.NewDevopsClient(s.DevopsOptions)
+		if err != nil {
+			return fmt.Errorf("failed to connect jenkins, please check jenkins status, error: %v", err)
+		}
+	}
+
+	var ldapClient ldapclient.Interface
+	// when there is no ldapOption, we set ldapClient as nil, which means we don't need to sync user info into ldap.
+	if s.LdapOptions != nil && len(s.LdapOptions.Host) != 0 {
+		if s.LdapOptions.Host == ldapclient.FAKE_HOST { // for debug only
+			ldapClient = ldapclient.NewSimpleLdap()
+		} else {
+			ldapClient, err = ldapclient.NewLdapClient(s.LdapOptions, ctx.Done())
+			if err != nil {
+				return fmt.Errorf("failed to connect to ldap service, please check ldap status, error: %v", err)
+			}
+		}
+	} else {
+		klog.Warning("ks-controller-manager starts without ldap provided, it will not sync user into ldap")
+	}
+
+	var s3Client s3.Interface
+	if s.S3Options != nil && len(s.S3Options.Endpoint) != 0 {
+		s3Client, err = s3.NewS3Client(s.S3Options)
+		if err != nil {
+			return fmt.Errorf("failed to connect to s3, please check s3 service status, error: %v", err)
+		}
+	}
+
+	informerFactory := informers.NewInformerFactories(
+		kubernetesClient.Kubernetes(),
+		kubernetesClient.KubeSphere(),
+		kubernetesClient.Istio(),
+		kubernetesClient.Snapshot(),
+		kubernetesClient.ApiExtensions(),
+		kubernetesClient.Prometheus())
+
+	mgrOptions := manager.Options{
+		CertDir: s.WebhookCertDir,
+		Port:    8443,
+	}
+
+	if s.LeaderElect {
+		mgrOptions = manager.Options{
+			CertDir:                 s.WebhookCertDir,
+			Port:                    8443,
+			LeaderElection:          s.LeaderElect,
+			LeaderElectionNamespace: "kubesphere-system",
+			LeaderElectionID:        "ks-controller-manager-leader-election",
+			LeaseDuration:           &s.LeaderElection.LeaseDuration,
+			RetryPeriod:             &s.LeaderElection.RetryPeriod,
+			RenewDeadline:           &s.LeaderElection.RenewDeadline,
+		}
+	}
+
+	klog.V(0).Info("setting up manager")
+	ctrl.SetLogger(klogr.New())
+	// Use 8443 instead of 443 cause we need root permission to bind port 443
+	mgr, err := manager.New(kubernetesClient.Config(), mgrOptions)
+	if err != nil {
+		klog.Fatalf("unable to set up overall controller manager: %v", err)
+	}
+
+	if err = apis.AddToScheme(mgr.GetScheme()); err != nil {
+		klog.Fatalf("unable add APIs to scheme: %v", err)
+	}
+
+	// register common meta types into schemas.
+	metav1.AddToGroupVersion(mgr.GetScheme(), metav1.SchemeGroupVersion)
+
+	kubeconfigClient := kubeconfig.NewOperator(kubernetesClient.Kubernetes(),
+		informerFactory.KubernetesSharedInformerFactory().Core().V1().ConfigMaps().Lister(),
+		kubernetesClient.Config())
+	userController := user.Reconciler{
+		MultiClusterEnabled:     s.MultiClusterOptions.Enable,
+		MaxConcurrentReconciles: 4,
+		LdapClient:              ldapClient,
+		DevopsClient:            devopsClient,
+		KubeconfigClient:        kubeconfigClient,
+		AuthenticationOptions:   s.AuthenticationOptions,
+	}
+
+	if err = userController.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create user controller: %v", err)
+	}
+
+	workspaceTemplateReconciler := &workspacetemplate.Reconciler{MultiClusterEnabled: s.MultiClusterOptions.Enable}
+	if err = workspaceTemplateReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create workspace template controller: %v", err)
+	}
+
+	workspaceReconciler := &workspace.Reconciler{}
+	if err = workspaceReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create workspace controller: %v", err)
+	}
+
+	workspaceRoleReconciler := &workspacerole.Reconciler{MultiClusterEnabled: s.MultiClusterOptions.Enable}
+	if err = workspaceRoleReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create workspace role controller: %v", err)
+	}
+
+	workspaceRoleBindingReconciler := &workspacerolebinding.Reconciler{MultiClusterEnabled: s.MultiClusterOptions.Enable}
+	if err = workspaceRoleBindingReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create workspace role binding controller: %v", err)
+	}
+
+	namespaceReconciler := &namespace.Reconciler{}
+	if err = namespaceReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create namespace controller: %v", err)
+	}
+
+	err = helmrepo.Add(mgr)
+	if err != nil {
+		klog.Fatal("Unable to create helm repo controller")
+	}
+
+	err = helmcategory.Add(mgr)
+	if err != nil {
+		klog.Fatal("Unable to create helm category controller")
+	}
+
+	var opS3Client s3.Interface
+	if !s.OpenPitrixOptions.AppStoreConfIsEmpty() {
+		opS3Client, err = s3.NewS3Client(s.OpenPitrixOptions.S3Options)
+		if err != nil {
+			klog.Fatalf("failed to connect to s3, please check openpitrix s3 service status, error: %v", err)
+		}
+		err = (&helmapplication.ReconcileHelmApplication{}).SetupWithManager(mgr)
+		if err != nil {
+			klog.Fatalf("Unable to create helm application controller, error: %s", err)
+		}
+
+		err = (&helmapplication.ReconcileHelmApplicationVersion{}).SetupWithManager(mgr)
+		if err != nil {
+			klog.Fatalf("Unable to create helm application version controller, error: %s ", err)
+		}
+	}
+
+	err = (&helmrelease.ReconcileHelmRelease{
+		// nil interface is valid value.
+		StorageClient:      opS3Client,
+		KsFactory:          informerFactory.KubeSphereSharedInformerFactory(),
+		MultiClusterEnable: s.MultiClusterOptions.Enable,
+		WaitTime:           s.OpenPitrixOptions.ReleaseControllerOptions.WaitTime,
+		MaxConcurrent:      s.OpenPitrixOptions.ReleaseControllerOptions.MaxConcurrent,
+		StopChan:           ctx.Done(),
+	}).SetupWithManager(mgr)
+
+	if err != nil {
+		klog.Fatalf("Unable to create helm release controller, error: %s", err)
+	}
+
+	selector, _ := labels.Parse(s.ApplicationSelector)
+	applicationReconciler := &application.ApplicationReconciler{
+		Scheme:              mgr.GetScheme(),
+		Client:              mgr.GetClient(),
+		Mapper:              mgr.GetRESTMapper(),
+		ApplicationSelector: selector,
+	}
+	if err = applicationReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create application controller: %v", err)
+	}
+
+	saReconciler := &serviceaccount.Reconciler{}
+	if err = saReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create ServiceAccount controller: %v", err)
+	}
+
+	resourceQuotaReconciler := quota.Reconciler{}
+	if err := resourceQuotaReconciler.SetupWithManager(mgr, quota.DefaultMaxConcurrentReconciles, quota.DefaultResyncPeriod, informerFactory.KubernetesSharedInformerFactory()); err != nil {
+		klog.Fatalf("Unable to create ResourceQuota controller: %v", err)
+	}
+
+	if !s.GatewayOptions.IsEmpty() {
+		helmReconciler := helm.Reconciler{GatewayOptions: s.GatewayOptions}
+		if err := helmReconciler.SetupWithManager(mgr); err != nil {
+			klog.Fatalf("Unable to create helm controller: %v", err)
+		}
+	}
+
+	// TODO(jeff): refactor config with CRD
+	servicemeshEnabled := s.ServiceMeshOptions != nil && len(s.ServiceMeshOptions.IstioPilotHost) != 0
+	if err = addControllers(mgr,
+		kubernetesClient,
+		informerFactory,
+		devopsClient,
+		s3Client,
+		ldapClient,
+		s.KubernetesOptions,
+		s.AuthenticationOptions,
+		s.MultiClusterOptions,
+		s.NetworkOptions,
+		servicemeshEnabled,
+		s.AuthenticationOptions.KubectlImage, ctx.Done()); err != nil {
+		klog.Fatalf("unable to register controllers to the manager: %v", err)
+	}
+
+	// Start cache data after all informer is registered
+	klog.V(0).Info("Starting cache resource from apiserver...")
+	informerFactory.Start(ctx.Done())
+
+	// Setup webhooks
+	klog.V(2).Info("setting up webhook server")
+	hookServer := mgr.GetWebhookServer()
+
+	klog.V(2).Info("registering webhooks to the webhook server")
+	hookServer.Register("/validate-email-iam-kubesphere-io-v1alpha2", &webhook.Admission{Handler: &user.EmailValidator{Client: mgr.GetClient()}})
+	hookServer.Register("/validate-network-kubesphere-io-v1alpha1", &webhook.Admission{Handler: &webhooks.ValidatingHandler{C: mgr.GetClient()}})
+	hookServer.Register("/mutate-network-kubesphere-io-v1alpha1", &webhook.Admission{Handler: &webhooks.MutatingHandler{C: mgr.GetClient()}})
+
+	resourceQuotaAdmission, err := quota.NewResourceQuotaAdmission(mgr.GetClient(), mgr.GetScheme())
+	if err != nil {
+		klog.Fatalf("unable to create resource quota admission: %v", err)
+	}
+	hookServer.Register("/validate-quota-kubesphere-io-v1alpha2", &webhook.Admission{Handler: resourceQuotaAdmission})
+
+	klog.V(2).Info("registering metrics to the webhook server")
+	// Add an extra metric endpoint, so we can use the the same metric definition with ks-apiserver
+	// /kapis/metrics is independent of controller-manager's built-in /metrics
+	mgr.AddMetricsExtraHandler("/kapis/metrics", metrics.Handler())
+
+	klog.V(0).Info("Starting the controllers.")
+	if err = mgr.Start(ctx); err != nil {
+		klog.Fatalf("unable to run the manager: %v", err)
+	}
+
+	return nil
+}

cmd/controller-manager/controller-manager.go

@@ -0,0 +1,31 @@
+/*
+Copyright 2020 KubeSphere Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	"os"
+
+	"kubesphere.io/kubesphere/cmd/controller-manager/app"
+)
+
+func main() {
+	command := app.NewControllerManagerCommand()
+
+	if err := command.Execute(); err != nil {
+		os.Exit(1)
+	}
+}

cmd/ks-apiserver/apiserver.go

@@ -0,0 +1,32 @@
+/*
+Copyright 2019 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	"log"
+
+	"kubesphere.io/kubesphere/cmd/ks-apiserver/app"
+)
+
+func main() {
+
+	cmd := app.NewAPIServerCommand()
+
+	if err := cmd.Execute(); err != nil {
+		log.Fatalln(err)
+	}
+}

cmd/ks-apiserver/app/options/options.go

@@ -0,0 +1,251 @@
+/*
+Copyright 2020 KubeSphere Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package options
+
+import (
+	"crypto/tls"
+	"flag"
+	"fmt"
+
+	"kubesphere.io/kubesphere/pkg/apiserver/authentication/token"
+
+	"k8s.io/client-go/kubernetes/scheme"
+	cliflag "k8s.io/component-base/cli/flag"
+	"k8s.io/klog"
+	runtimecache "sigs.k8s.io/controller-runtime/pkg/cache"
+	runtimeclient "sigs.k8s.io/controller-runtime/pkg/client"
+
+	"kubesphere.io/kubesphere/pkg/apis"
+	"kubesphere.io/kubesphere/pkg/apiserver"
+	apiserverconfig "kubesphere.io/kubesphere/pkg/apiserver/config"
+	"kubesphere.io/kubesphere/pkg/informers"
+	genericoptions "kubesphere.io/kubesphere/pkg/server/options"
+	"kubesphere.io/kubesphere/pkg/simple/client/alerting"
+	auditingclient "kubesphere.io/kubesphere/pkg/simple/client/auditing/elasticsearch"
+	"kubesphere.io/kubesphere/pkg/simple/client/cache"
+
+	"net/http"
+	"strings"
+
+	"kubesphere.io/kubesphere/pkg/simple/client/devops/jenkins"
+	eventsclient "kubesphere.io/kubesphere/pkg/simple/client/events/elasticsearch"
+	"kubesphere.io/kubesphere/pkg/simple/client/k8s"
+	esclient "kubesphere.io/kubesphere/pkg/simple/client/logging/elasticsearch"
+	"kubesphere.io/kubesphere/pkg/simple/client/monitoring/metricsserver"
+	"kubesphere.io/kubesphere/pkg/simple/client/monitoring/prometheus"
+	"kubesphere.io/kubesphere/pkg/simple/client/s3"
+	fakes3 "kubesphere.io/kubesphere/pkg/simple/client/s3/fake"
+	"kubesphere.io/kubesphere/pkg/simple/client/sonarqube"
+)
+
+type ServerRunOptions struct {
+	ConfigFile              string
+	GenericServerRunOptions *genericoptions.ServerRunOptions
+	*apiserverconfig.Config
+
+	//
+	DebugMode bool
+}
+
+func NewServerRunOptions() *ServerRunOptions {
+	s := &ServerRunOptions{
+		GenericServerRunOptions: genericoptions.NewServerRunOptions(),
+		Config:                  apiserverconfig.New(),
+	}
+
+	return s
+}
+
+func (s *ServerRunOptions) Flags() (fss cliflag.NamedFlagSets) {
+	fs := fss.FlagSet("generic")
+	fs.BoolVar(&s.DebugMode, "debug", false, "Don't enable this if you don't know what it means.")
+	s.GenericServerRunOptions.AddFlags(fs, s.GenericServerRunOptions)
+	s.KubernetesOptions.AddFlags(fss.FlagSet("kubernetes"), s.KubernetesOptions)
+	s.AuthenticationOptions.AddFlags(fss.FlagSet("authentication"), s.AuthenticationOptions)
+	s.AuthorizationOptions.AddFlags(fss.FlagSet("authorization"), s.AuthorizationOptions)
+	s.DevopsOptions.AddFlags(fss.FlagSet("devops"), s.DevopsOptions)
+	s.SonarQubeOptions.AddFlags(fss.FlagSet("sonarqube"), s.SonarQubeOptions)
+	s.RedisOptions.AddFlags(fss.FlagSet("redis"), s.RedisOptions)
+	s.S3Options.AddFlags(fss.FlagSet("s3"), s.S3Options)
+	s.OpenPitrixOptions.AddFlags(fss.FlagSet("openpitrix"), s.OpenPitrixOptions)
+	s.NetworkOptions.AddFlags(fss.FlagSet("network"), s.NetworkOptions)
+	s.ServiceMeshOptions.AddFlags(fss.FlagSet("servicemesh"), s.ServiceMeshOptions)
+	s.MonitoringOptions.AddFlags(fss.FlagSet("monitoring"), s.MonitoringOptions)
+	s.LoggingOptions.AddFlags(fss.FlagSet("logging"), s.LoggingOptions)
+	s.MultiClusterOptions.AddFlags(fss.FlagSet("multicluster"), s.MultiClusterOptions)
+	s.EventsOptions.AddFlags(fss.FlagSet("events"), s.EventsOptions)
+	s.AuditingOptions.AddFlags(fss.FlagSet("auditing"), s.AuditingOptions)
+	s.AlertingOptions.AddFlags(fss.FlagSet("alerting"), s.AlertingOptions)
+
+	fs = fss.FlagSet("klog")
+	local := flag.NewFlagSet("klog", flag.ExitOnError)
+	klog.InitFlags(local)
+	local.VisitAll(func(fl *flag.Flag) {
+		fl.Name = strings.Replace(fl.Name, "_", "-", -1)
+		fs.AddGoFlag(fl)
+	})
+
+	return fss
+}
+
+const fakeInterface string = "FAKE"
+
+// NewAPIServer creates an APIServer instance using given options
+func (s *ServerRunOptions) NewAPIServer(stopCh <-chan struct{}) (*apiserver.APIServer, error) {
+	apiServer := &apiserver.APIServer{
+		Config: s.Config,
+	}
+
+	kubernetesClient, err := k8s.NewKubernetesClient(s.KubernetesOptions)
+	if err != nil {
+		return nil, err
+	}
+	apiServer.KubernetesClient = kubernetesClient
+
+	informerFactory := informers.NewInformerFactories(kubernetesClient.Kubernetes(), kubernetesClient.KubeSphere(),
+		kubernetesClient.Istio(), kubernetesClient.Snapshot(), kubernetesClient.ApiExtensions(), kubernetesClient.Prometheus())
+	apiServer.InformerFactory = informerFactory
+
+	if s.MonitoringOptions == nil || len(s.MonitoringOptions.Endpoint) == 0 {
+		return nil, fmt.Errorf("moinitoring service address in configuration MUST not be empty, please check configmap/kubesphere-config in kubesphere-system namespace")
+	} else {
+		monitoringClient, err := prometheus.NewPrometheus(s.MonitoringOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connect to prometheus, please check prometheus status, error: %v", err)
+		}
+		apiServer.MonitoringClient = monitoringClient
+	}
+
+	apiServer.MetricsClient = metricsserver.NewMetricsClient(kubernetesClient.Kubernetes(), s.KubernetesOptions)
+
+	if s.LoggingOptions.Host != "" {
+		loggingClient, err := esclient.NewClient(s.LoggingOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connect to elasticsearch, please check elasticsearch status, error: %v", err)
+		}
+		apiServer.LoggingClient = loggingClient
+	}
+
+	if s.S3Options.Endpoint != "" {
+		if s.S3Options.Endpoint == fakeInterface && s.DebugMode {
+			apiServer.S3Client = fakes3.NewFakeS3()
+		} else {
+			s3Client, err := s3.NewS3Client(s.S3Options)
+			if err != nil {
+				return nil, fmt.Errorf("failed to connect to s3, please check s3 service status, error: %v", err)
+			}
+			apiServer.S3Client = s3Client
+		}
+	}
+
+	if s.DevopsOptions.Host != "" {
+		devopsClient, err := jenkins.NewDevopsClient(s.DevopsOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connect to jenkins, please check jenkins status, error: %v", err)
+		}
+		apiServer.DevopsClient = devopsClient
+	}
+
+	if s.SonarQubeOptions.Host != "" {
+		sonarClient, err := sonarqube.NewSonarQubeClient(s.SonarQubeOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connecto to sonarqube, please check sonarqube status, error: %v", err)
+		}
+		apiServer.SonarClient = sonarqube.NewSonar(sonarClient.SonarQube())
+	}
+
+	var cacheClient cache.Interface
+	if s.RedisOptions != nil && len(s.RedisOptions.Host) != 0 {
+		if s.RedisOptions.Host == fakeInterface && s.DebugMode {
+			apiServer.CacheClient = cache.NewSimpleCache()
+		} else {
+			cacheClient, err = cache.NewRedisClient(s.RedisOptions, stopCh)
+			if err != nil {
+				return nil, fmt.Errorf("failed to connect to redis service, please check redis status, error: %v", err)
+			}
+			apiServer.CacheClient = cacheClient
+		}
+	} else {
+		klog.Warning("ks-apiserver starts without redis provided, it will use in memory cache. " +
+			"This may cause inconsistencies when running ks-apiserver with multiple replicas.")
+		apiServer.CacheClient = cache.NewSimpleCache()
+	}
+
+	if s.EventsOptions.Host != "" {
+		eventsClient, err := eventsclient.NewClient(s.EventsOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connect to elasticsearch, please check elasticsearch status, error: %v", err)
+		}
+		apiServer.EventsClient = eventsClient
+	}
+
+	if s.AuditingOptions.Host != "" {
+		auditingClient, err := auditingclient.NewClient(s.AuditingOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connect to elasticsearch, please check elasticsearch status, error: %v", err)
+		}
+		apiServer.AuditingClient = auditingClient
+	}
+
+	if s.AlertingOptions != nil && (s.AlertingOptions.PrometheusEndpoint != "" || s.AlertingOptions.ThanosRulerEndpoint != "") {
+		alertingClient, err := alerting.NewRuleClient(s.AlertingOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to init alerting client: %v", err)
+		}
+		apiServer.AlertingClient = alertingClient
+	}
+
+	server := &http.Server{
+		Addr: fmt.Sprintf(":%d", s.GenericServerRunOptions.InsecurePort),
+	}
+
+	if s.GenericServerRunOptions.SecurePort != 0 {
+		certificate, err := tls.LoadX509KeyPair(s.GenericServerRunOptions.TlsCertFile, s.GenericServerRunOptions.TlsPrivateKey)
+		if err != nil {
+			return nil, err
+		}
+
+		server.TLSConfig = &tls.Config{
+			Certificates: []tls.Certificate{certificate},
+		}
+		server.Addr = fmt.Sprintf(":%d", s.GenericServerRunOptions.SecurePort)
+	}
+
+	sch := scheme.Scheme
+	if err := apis.AddToScheme(sch); err != nil {
+		klog.Fatalf("unable add APIs to scheme: %v", err)
+	}
+
+	apiServer.RuntimeCache, err = runtimecache.New(apiServer.KubernetesClient.Config(), runtimecache.Options{Scheme: sch})
+	if err != nil {
+		klog.Fatalf("unable to create controller runtime cache: %v", err)
+	}
+
+	apiServer.RuntimeClient, err = runtimeclient.New(apiServer.KubernetesClient.Config(), runtimeclient.Options{Scheme: sch})
+	if err != nil {
+		klog.Fatalf("unable to create controller runtime client: %v", err)
+	}
+
+	apiServer.Issuer, err = token.NewIssuer(s.AuthenticationOptions)
+	if err != nil {
+		klog.Fatalf("unable to create issuer: %v", err)
+	}
+
+	apiServer.Server = server
+
+	return apiServer, nil
+}

cmd/ks-apiserver/app/options/validation.go

@@ -0,0 +1,41 @@
+/*
+Copyright 2020 KubeSphere Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package options
+
+// Validate validates server run options, to find
+// options' misconfiguration
+func (s *ServerRunOptions) Validate() []error {
+	var errors []error
+
+	errors = append(errors, s.GenericServerRunOptions.Validate()...)
+	errors = append(errors, s.DevopsOptions.Validate()...)
+	errors = append(errors, s.KubernetesOptions.Validate()...)
+	errors = append(errors, s.ServiceMeshOptions.Validate()...)
+	errors = append(errors, s.MonitoringOptions.Validate()...)
+	errors = append(errors, s.SonarQubeOptions.Validate()...)
+	errors = append(errors, s.S3Options.Validate()...)
+	errors = append(errors, s.OpenPitrixOptions.Validate()...)
+	errors = append(errors, s.NetworkOptions.Validate()...)
+	errors = append(errors, s.LoggingOptions.Validate()...)
+	errors = append(errors, s.AuthenticationOptions.Validate()...)
+	errors = append(errors, s.AuthorizationOptions.Validate()...)
+	errors = append(errors, s.EventsOptions.Validate()...)
+	errors = append(errors, s.AuditingOptions.Validate()...)
+	errors = append(errors, s.AlertingOptions.Validate()...)
+
+	return errors
+}

cmd/ks-apiserver/app/server.go

@@ -0,0 +1,104 @@
+/*
+Copyright 2019 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package app
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/spf13/cobra"
+	utilerrors "k8s.io/apimachinery/pkg/util/errors"
+	cliflag "k8s.io/component-base/cli/flag"
+	"k8s.io/klog"
+
+	"sigs.k8s.io/controller-runtime/pkg/manager/signals"
+
+	"kubesphere.io/kubesphere/cmd/ks-apiserver/app/options"
+	apiserverconfig "kubesphere.io/kubesphere/pkg/apiserver/config"
+	"kubesphere.io/kubesphere/pkg/utils/term"
+	"kubesphere.io/kubesphere/pkg/version"
+)
+
+func NewAPIServerCommand() *cobra.Command {
+	s := options.NewServerRunOptions()
+
+	// Load configuration from file
+	conf, err := apiserverconfig.TryLoadFromDisk()
+	if err == nil {
+		s = &options.ServerRunOptions{
+			GenericServerRunOptions: s.GenericServerRunOptions,
+			Config:                  conf,
+		}
+	} else {
+		klog.Fatal("Failed to load configuration from disk", err)
+	}
+
+	cmd := &cobra.Command{
+		Use: "ks-apiserver",
+		Long: `The KubeSphere API server validates and configures data for the API objects. 
+The API Server services REST operations and provides the frontend to the
+cluster's shared state through which all other components interact.`,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if errs := s.Validate(); len(errs) != 0 {
+				return utilerrors.NewAggregate(errs)
+			}
+
+			return Run(s, signals.SetupSignalHandler())
+		},
+		SilenceUsage: true,
+	}
+
+	fs := cmd.Flags()
+	namedFlagSets := s.Flags()
+	for _, f := range namedFlagSets.FlagSets {
+		fs.AddFlagSet(f)
+	}
+
+	usageFmt := "Usage:\n  %s\n"
+	cols, _, _ := term.TerminalSize(cmd.OutOrStdout())
+	cmd.SetHelpFunc(func(cmd *cobra.Command, args []string) {
+		fmt.Fprintf(cmd.OutOrStdout(), "%s\n\n"+usageFmt, cmd.Long, cmd.UseLine())
+		cliflag.PrintSections(cmd.OutOrStdout(), namedFlagSets, cols)
+	})
+
+	versionCmd := &cobra.Command{
+		Use:   "version",
+		Short: "Print the version of KubeSphere ks-apiserver",
+		Run: func(cmd *cobra.Command, args []string) {
+			cmd.Println(version.Get())
+		},
+	}
+
+	cmd.AddCommand(versionCmd)
+
+	return cmd
+}
+
+func Run(s *options.ServerRunOptions, ctx context.Context) error {
+
+	apiserver, err := s.NewAPIServer(ctx.Done())
+	if err != nil {
+		return err
+	}
+
+	err = apiserver.PrepareRun(ctx.Done())
+	if err != nil {
+		return err
+	}
+
+	return apiserver.Run(ctx)
+}

cmd/kubesphere.go

@@ -1,40 +0,0 @@
-/*
-Copyright 2018 The KubeSphere Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package main
-
-import (
-	"github.com/spf13/pflag"
-
-	"kubesphere.io/kubesphere/pkg/app"
-	"kubesphere.io/kubesphere/pkg/logs"
-	"kubesphere.io/kubesphere/pkg/options"
-	"kubesphere.io/kubesphere/pkg/version"
-)
-
-func main() {
-
-	options.AddFlags(pflag.CommandLine)
-
-	pflag.Parse()
-	logs.InitLogs()
-
-	defer logs.FlushLogs()
-
-	version.PrintAndExitIfRequested()
-
-	app.Run()
-}

config/crds/app_v1beta1_application.yaml

@@ -0,0 +1,532 @@
+# Copyright 2020 The Kubernetes Authors.
+# SPDX-License-Identifier: Apache-2.0
+
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    api-approved.kubernetes.io: https://github.com/kubernetes-sigs/application/pull/2
+    controller-gen.kubebuilder.io/version: v0.4.0
+  creationTimestamp: null
+  name: applications.app.k8s.io
+spec:
+  group: app.k8s.io
+  names:
+    categories:
+    - all
+    kind: Application
+    listKind: ApplicationList
+    plural: applications
+    shortNames:
+    - app
+    singular: application
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - description: The type of the application
+      jsonPath: .spec.descriptor.type
+      name: Type
+      type: string
+    - description: The creation date
+      jsonPath: .spec.descriptor.version
+      name: Version
+      type: string
+    - description: The application object owns the matched resources
+      jsonPath: .spec.addOwnerRef
+      name: Owner
+      type: boolean
+    - description: Numbers of components ready
+      jsonPath: .status.componentsReady
+      name: Ready
+      type: string
+    - description: The creation date
+      jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1beta1
+    schema:
+      openAPIV3Schema:
+        description: Application is the Schema for the applications API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: ApplicationSpec defines the specification for an Application.
+            properties:
+              addOwnerRef:
+                description: AddOwnerRef objects - flag to indicate if we need to
+                  add OwnerRefs to matching objects Matching is done by using Selector
+                  to query all ComponentGroupKinds
+                type: boolean
+              assemblyPhase:
+                description: AssemblyPhase represents the current phase of the application's
+                  assembly. An empty value is equivalent to "Succeeded".
+                type: string
+              componentKinds:
+                description: ComponentGroupKinds is a list of Kinds for Application's
+                  components (e.g. Deployments, Pods, Services, CRDs). It can be used
+                  in conjunction with the Application's Selector to list or watch
+                  the Applications components.
+                items:
+                  description: GroupKind specifies a Group and a Kind, but does not
+                    force a version.  This is useful for identifying concepts during
+                    lookup stages without having partially valid types
+                  properties:
+                    group:
+                      type: string
+                    kind:
+                      type: string
+                  required:
+                  - group
+                  - kind
+                  type: object
+                type: array
+              descriptor:
+                description: Descriptor regroups information and metadata about an
+                  application.
+                properties:
+                  description:
+                    description: Description is a brief string description of the
+                      Application.
+                    type: string
+                  icons:
+                    description: Icons is an optional list of icons for an application.
+                      Icon information includes the source, size, and mime type.
+                    items:
+                      description: ImageSpec contains information about an image used
+                        as an icon.
+                      properties:
+                        size:
+                          description: (optional) The size of the image in pixels
+                            (e.g., 25x25).
+                          type: string
+                        src:
+                          description: The source for image represented as either
+                            an absolute URL to the image or a Data URL containing
+                            the image. Data URLs are defined in RFC 2397.
+                          type: string
+                        type:
+                          description: (optional) The mine type of the image (e.g.,
+                            "image/png").
+                          type: string
+                      required:
+                      - src
+                      type: object
+                    type: array
+                  keywords:
+                    description: Keywords is an optional list of key words associated
+                      with the application (e.g. MySQL, RDBMS, database).
+                    items:
+                      type: string
+                    type: array
+                  links:
+                    description: Links are a list of descriptive URLs intended to
+                      be used to surface additional documentation, dashboards, etc.
+                    items:
+                      description: Link contains information about an URL to surface
+                        documentation, dashboards, etc.
+                      properties:
+                        description:
+                          description: Description is human readable content explaining
+                            the purpose of the link.
+                          type: string
+                        url:
+                          description: Url typically points at a website address.
+                          type: string
+                      type: object
+                    type: array
+                  maintainers:
+                    description: Maintainers is an optional list of maintainers of
+                      the application. The maintainers in this list maintain the the
+                      source code, images, and package for the application.
+                    items:
+                      description: ContactData contains information about an individual
+                        or organization.
+                      properties:
+                        email:
+                          description: Email is the email address.
+                          type: string
+                        name:
+                          description: Name is the descriptive name.
+                          type: string
+                        url:
+                          description: Url could typically be a website address.
+                          type: string
+                      type: object
+                    type: array
+                  notes:
+                    description: Notes contain a human readable snippets intended
+                      as a quick start for the users of the Application. CommonMark
+                      markdown syntax may be used for rich text representation.
+                    type: string
+                  owners:
+                    description: Owners is an optional list of the owners of the installed
+                      application. The owners of the application should be contacted
+                      in the event of a planned or unplanned disruption affecting
+                      the application.
+                    items:
+                      description: ContactData contains information about an individual
+                        or organization.
+                      properties:
+                        email:
+                          description: Email is the email address.
+                          type: string
+                        name:
+                          description: Name is the descriptive name.
+                          type: string
+                        url:
+                          description: Url could typically be a website address.
+                          type: string
+                      type: object
+                    type: array
+                  type:
+                    description: Type is the type of the application (e.g. WordPress,
+                      MySQL, Cassandra).
+                    type: string
+                  version:
+                    description: Version is an optional version indicator for the
+                      Application.
+                    type: string
+                type: object
+              info:
+                description: Info contains human readable key,value pairs for the
+                  Application.
+                items:
+                  description: InfoItem is a human readable key,value pair containing
+                    important information about how to access the Application.
+                  properties:
+                    name:
+                      description: Name is a human readable title for this piece of
+                        information.
+                      type: string
+                    type:
+                      description: Type of the value for this InfoItem.
+                      type: string
+                    value:
+                      description: Value is human readable content.
+                      type: string
+                    valueFrom:
+                      description: ValueFrom defines a reference to derive the value
+                        from another source.
+                      properties:
+                        configMapKeyRef:
+                          description: Selects a key of a ConfigMap.
+                          properties:
+                            apiVersion:
+                              description: API version of the referent.
+                              type: string
+                            fieldPath:
+                              description: 'If referring to a piece of an object instead
+                                of an entire object, this string should contain a
+                                valid JSON/Go field access statement, such as desiredState.manifest.containers[2].
+                                For example, if the object reference is to a container
+                                within a pod, this would take on a value like: "spec.containers{name}"
+                                (where "name" refers to the name of the container
+                                that triggered the event) or if no container name
+                                is specified "spec.containers[2]" (container with
+                                index 2 in this pod). This syntax is chosen only to
+                                have some well-defined way of referencing a part of
+                                an object. TODO: this design is not final and this
+                                field is subject to change in the future.'
+                              type: string
+                            key:
+                              description: The key to select.
+                              type: string
+                            kind:
+                              description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                              type: string
+                            name:
+                              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                              type: string
+                            namespace:
+                              description: 'Namespace of the referent. More info:
+                                https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                              type: string
+                            resourceVersion:
+                              description: 'Specific resourceVersion to which this
+                                reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                              type: string
+                            uid:
+                              description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                              type: string
+                          type: object
+                        ingressRef:
+                          description: Select an Ingress.
+                          properties:
+                            apiVersion:
+                              description: API version of the referent.
+                              type: string
+                            fieldPath:
+                              description: 'If referring to a piece of an object instead
+                                of an entire object, this string should contain a
+                                valid JSON/Go field access statement, such as desiredState.manifest.containers[2].
+                                For example, if the object reference is to a container
+                                within a pod, this would take on a value like: "spec.containers{name}"
+                                (where "name" refers to the name of the container
+                                that triggered the event) or if no container name
+                                is specified "spec.containers[2]" (container with
+                                index 2 in this pod). This syntax is chosen only to
+                                have some well-defined way of referencing a part of
+                                an object. TODO: this design is not final and this
+                                field is subject to change in the future.'
+                              type: string
+                            host:
+                              description: The optional host to select.
+                              type: string
+                            kind:
+                              description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                              type: string
+                            name:
+                              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                              type: string
+                            namespace:
+                              description: 'Namespace of the referent. More info:
+                                https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                              type: string
+                            path:
+                              description: The optional HTTP path.
+                              type: string
+                            protocol:
+                              description: Protocol for the ingress
+                              type: string
+                            resourceVersion:
+                              description: 'Specific resourceVersion to which this
+                                reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                              type: string
+                            uid:
+                              description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                              type: string
+                          type: object
+                        secretKeyRef:
+                          description: Selects a key of a Secret.
+                          properties:
+                            apiVersion:
+                              description: API version of the referent.
+                              type: string
+                            fieldPath:
+                              description: 'If referring to a piece of an object instead
+                                of an entire object, this string should contain a
+                                valid JSON/Go field access statement, such as desiredState.manifest.containers[2].
+                                For example, if the object reference is to a container
+                                within a pod, this would take on a value like: "spec.containers{name}"
+                                (where "name" refers to the name of the container
+                                that triggered the event) or if no container name
+                                is specified "spec.containers[2]" (container with
+                                index 2 in this pod). This syntax is chosen only to
+                                have some well-defined way of referencing a part of
+                                an object. TODO: this design is not final and this
+                                field is subject to change in the future.'
+                              type: string
+                            key:
+                              description: The key to select.
+                              type: string
+                            kind:
+                              description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                              type: string
+                            name:
+                              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                              type: string
+                            namespace:
+                              description: 'Namespace of the referent. More info:
+                                https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                              type: string
+                            resourceVersion:
+                              description: 'Specific resourceVersion to which this
+                                reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                              type: string
+                            uid:
+                              description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                              type: string
+                          type: object
+                        serviceRef:
+                          description: Select a Service.
+                          properties:
+                            apiVersion:
+                              description: API version of the referent.
+                              type: string
+                            fieldPath:
+                              description: 'If referring to a piece of an object instead
+                                of an entire object, this string should contain a
+                                valid JSON/Go field access statement, such as desiredState.manifest.containers[2].
+                                For example, if the object reference is to a container
+                                within a pod, this would take on a value like: "spec.containers{name}"
+                                (where "name" refers to the name of the container
+                                that triggered the event) or if no container name
+                                is specified "spec.containers[2]" (container with
+                                index 2 in this pod). This syntax is chosen only to
+                                have some well-defined way of referencing a part of
+                                an object. TODO: this design is not final and this
+                                field is subject to change in the future.'
+                              type: string
+                            kind:
+                              description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                              type: string
+                            name:
+                              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                              type: string
+                            namespace:
+                              description: 'Namespace of the referent. More info:
+                                https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                              type: string
+                            path:
+                              description: The optional HTTP path.
+                              type: string
+                            port:
+                              description: The optional port to select.
+                              format: int32
+                              type: integer
+                            protocol:
+                              description: Protocol for the service
+                              type: string
+                            resourceVersion:
+                              description: 'Specific resourceVersion to which this
+                                reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                              type: string
+                            uid:
+                              description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                              type: string
+                          type: object
+                        type:
+                          description: Type of source.
+                          type: string
+                      type: object
+                  type: object
+                type: array
+              selector:
+                description: 'Selector is a label query over kinds that created by
+                  the application. It must match the component objects'' labels. More
+                  info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors'
+                properties:
+                  matchExpressions:
+                    description: matchExpressions is a list of label selector requirements.
+                      The requirements are ANDed.
+                    items:
+                      description: A label selector requirement is a selector that
+                        contains values, a key, and an operator that relates the key
+                        and values.
+                      properties:
+                        key:
+                          description: key is the label key that the selector applies
+                            to.
+                          type: string
+                        operator:
+                          description: operator represents a key's relationship to
+                            a set of values. Valid operators are In, NotIn, Exists
+                            and DoesNotExist.
+                          type: string
+                        values:
+                          description: values is an array of string values. If the
+                            operator is In or NotIn, the values array must be non-empty.
+                            If the operator is Exists or DoesNotExist, the values
+                            array must be empty. This array is replaced during a strategic
+                            merge patch.
+                          items:
+                            type: string
+                          type: array
+                      required:
+                      - key
+                      - operator
+                      type: object
+                    type: array
+                  matchLabels:
+                    additionalProperties:
+                      type: string
+                    description: matchLabels is a map of {key,value} pairs. A single
+                      {key,value} in the matchLabels map is equivalent to an element
+                      of matchExpressions, whose key field is "key", the operator
+                      is "In", and the values array contains only "value". The requirements
+                      are ANDed.
+                    type: object
+                type: object
+            type: object
+          status:
+            description: ApplicationStatus defines controller's the observed state
+              of Application
+            properties:
+              components:
+                description: Object status array for all matching objects
+                items:
+                  description: ObjectStatus is a generic status holder for objects
+                  properties:
+                    group:
+                      description: Object group
+                      type: string
+                    kind:
+                      description: Kind of object
+                      type: string
+                    link:
+                      description: Link to object
+                      type: string
+                    name:
+                      description: Name of object
+                      type: string
+                    status:
+                      description: 'Status. Values: InProgress, Ready, Unknown'
+                      type: string
+                  type: object
+                type: array
+              componentsReady:
+                description: 'ComponentsReady: status of the components in the format
+                  ready/total'
+                type: string
+              conditions:
+                description: Conditions represents the latest state of the object
+                items:
+                  description: Condition describes the state of an object at a certain
+                    point.
+                  properties:
+                    lastTransitionTime:
+                      description: Last time the condition transitioned from one status
+                        to another.
+                      format: date-time
+                      type: string
+                    lastUpdateTime:
+                      description: Last time the condition was probed
+                      format: date-time
+                      type: string
+                    message:
+                      description: A human readable message indicating details about
+                        the transition.
+                      type: string
+                    reason:
+                      description: The reason for the condition's last transition.
+                      type: string
+                    status:
+                      description: Status of the condition, one of True, False, Unknown.
+                      type: string
+                    type:
+                      description: Type of condition.
+                      type: string
+                  required:
+                  - status
+                  - type
+                  type: object
+                type: array
+              observedGeneration:
+                description: ObservedGeneration is the most recent generation observed.
+                  It corresponds to the Object's generation, which is updated on mutation
+                  by the API Server.
+                format: int64
+                type: integer
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/application.kubesphere.io_helmapplications.yaml

@@ -0,0 +1,101 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: helmapplications.application.kubesphere.io
+spec:
+  group: application.kubesphere.io
+  names:
+    kind: HelmApplication
+    listKind: HelmApplicationList
+    plural: helmapplications
+    shortNames:
+    - happ
+    singular: helmapplication
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.name
+      name: application name
+      type: string
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: workspace
+      type: string
+    - jsonPath: .status.state
+      name: State
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: HelmApplication is the Schema for the helmapplications API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: HelmApplicationSpec defines the desired state of HelmApplication
+            properties:
+              abstraction:
+                description: info from frontend
+                type: string
+              appHome:
+                type: string
+              attachments:
+                description: attachments id
+                items:
+                  type: string
+                type: array
+              description:
+                description: description from chart's description or frontend
+                type: string
+              icon:
+                description: The attachment id of the icon
+                type: string
+              name:
+                description: the name of the helm application
+                type: string
+            required:
+            - name
+            type: object
+          status:
+            description: HelmApplicationStatus defines the observed state of HelmApplication
+            properties:
+              latestVersion:
+                description: If this application belong to appStore, latestVersion is the the latest version of the active application version. otherwise latestVersion is the latest version of all application version
+                type: string
+              state:
+                description: 'the state of the helm application: draft, submitted, passed, rejected, suspended, active'
+                type: string
+              statusTime:
+                format: date-time
+                type: string
+              updateTime:
+                format: date-time
+                type: string
+            required:
+            - statusTime
+            - updateTime
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/application.kubesphere.io_helmapplicationversions.yaml

@@ -0,0 +1,205 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: helmapplicationversions.application.kubesphere.io
+spec:
+  group: application.kubesphere.io
+  names:
+    kind: HelmApplicationVersion
+    listKind: HelmApplicationVersionList
+    plural: helmapplicationversions
+    shortNames:
+    - happver
+    singular: helmapplicationversion
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.name
+      name: application name
+      type: string
+    - jsonPath: .status.state
+      name: State
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: HelmApplicationVersion is the Schema for the helmapplicationversions API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: HelmApplicationVersionSpec defines the desired state of HelmApplicationVersion
+            properties:
+              annotations:
+                additionalProperties:
+                  type: string
+                description: Annotations are additional mappings uninterpreted by Helm, made available for inspection by other applications.
+                type: object
+              apiVersion:
+                description: The API Version of this chart.
+                type: string
+              appVersion:
+                description: The version of the application enclosed inside of this chart.
+                type: string
+              condition:
+                description: The condition to check to enable chart
+                type: string
+              created:
+                description: chart create time
+                format: date-time
+                type: string
+              data:
+                description: raw data of chart, it will !!!NOT!!! be save to etcd
+                format: byte
+                type: string
+              dataKey:
+                description: dataKey in the storage
+                type: string
+              dependencies:
+                description: Dependencies are a list of dependencies for a chart.
+                items:
+                  description: Dependency describes a chart upon which another chart depends. Dependencies can be used to express developer intent, or to capture the state of a chart.
+                  properties:
+                    alias:
+                      description: Alias usable alias to be used for the chart
+                      type: string
+                    condition:
+                      description: A yaml path that resolves to a boolean, used for enabling/disabling charts (e.g. subchart1.enabled )
+                      type: string
+                    enabled:
+                      description: Enabled bool determines if chart should be loaded
+                      type: boolean
+                    name:
+                      description: Name is the name of the dependency. This must mach the name in the dependency's Chart.yaml.
+                      type: string
+                    repository:
+                      description: The URL to the repository. Appending `index.yaml` to this string should result in a URL that can be used to fetch the repository index.
+                      type: string
+                    tags:
+                      description: Tags can be used to group charts for enabling/disabling together
+                      items:
+                        type: string
+                      type: array
+                    version:
+                      description: Version is the version (range) of this chart. A lock file will always produce a single version, while a dependency may contain a semantic version range.
+                      type: string
+                  required:
+                  - name
+                  - repository
+                  type: object
+                type: array
+              deprecated:
+                description: Whether or not this chart is deprecated
+                type: boolean
+              description:
+                description: A one-sentence description of the chart
+                type: string
+              digest:
+                description: chart digest
+                type: string
+              home:
+                description: The URL to a relevant project page, git repo, or contact person
+                type: string
+              icon:
+                description: The URL to an icon file.
+                type: string
+              keywords:
+                description: A list of string keywords
+                items:
+                  type: string
+                type: array
+              kubeVersion:
+                description: KubeVersion is a SemVer constraint specifying the version of Kubernetes required.
+                type: string
+              maintainers:
+                description: A list of name and URL/email address combinations for the maintainer(s)
+                items:
+                  description: Maintainer describes a Chart maintainer.
+                  properties:
+                    email:
+                      description: Email is an optional email address to contact the named maintainer
+                      type: string
+                    name:
+                      description: Name is a user name or organization name
+                      type: string
+                    url:
+                      description: URL is an optional URL to an address for the named maintainer
+                      type: string
+                  type: object
+                type: array
+              name:
+                description: The name of the chart
+                type: string
+              sources:
+                description: Source is the URL to the source code of this chart
+                items:
+                  type: string
+                type: array
+              tags:
+                description: The tags to check to enable chart
+                type: string
+              type:
+                description: 'Specifies the chart type: application or library'
+                type: string
+              urls:
+                description: chart url
+                items:
+                  type: string
+                type: array
+              version:
+                description: A SemVer 2 conformant version string of the chart
+                type: string
+            type: object
+          status:
+            description: HelmApplicationVersionStatus defines the observed state of HelmApplicationVersion
+            properties:
+              audit:
+                items:
+                  properties:
+                    message:
+                      description: audit message
+                      type: string
+                    operator:
+                      description: audit operator
+                      type: string
+                    operatorType:
+                      type: string
+                    state:
+                      description: 'audit state: submitted, passed, draft, active, rejected, suspended'
+                      type: string
+                    time:
+                      description: audit time
+                      format: date-time
+                      type: string
+                  required:
+                  - time
+                  type: object
+                type: array
+              state:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/application.kubesphere.io_helmcategories.yaml

@@ -0,0 +1,76 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: helmcategories.application.kubesphere.io
+spec:
+  group: application.kubesphere.io
+  names:
+    kind: HelmCategory
+    listKind: HelmCategoryList
+    plural: helmcategories
+    shortNames:
+    - hctg
+    singular: helmcategory
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.name
+      name: name
+      type: string
+    - jsonPath: .status.total
+      name: total
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: HelmCategory is the Schema for the helmcategories API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: HelmCategorySpec defines the desired state of HelmRepo
+            properties:
+              description:
+                description: info from frontend
+                type: string
+              locale:
+                type: string
+              name:
+                description: name of the category
+                type: string
+            required:
+            - name
+            type: object
+          status:
+            properties:
+              total:
+                description: total helmapplications belong to this category
+                type: integer
+            required:
+            - total
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/application.kubesphere.io_helmreleases.yaml

@@ -0,0 +1,145 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: helmreleases.application.kubesphere.io
+spec:
+  group: application.kubesphere.io
+  names:
+    kind: HelmRelease
+    listKind: HelmReleaseList
+    plural: helmreleases
+    shortNames:
+    - hrls
+    singular: helmrelease
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.name
+      name: Release Name
+      type: string
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: Workspace
+      type: string
+    - jsonPath: .metadata.labels.kubesphere\.io/cluster
+      name: Cluster
+      type: string
+    - jsonPath: .metadata.labels.kubesphere\.io/namespace
+      name: Namespace
+      type: string
+    - jsonPath: .status.state
+      name: State
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: HelmRelease is the Schema for the helmreleases API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: HelmReleaseSpec defines the desired state of HelmRelease
+            properties:
+              appId:
+                description: id of the helmapplication
+                type: string
+              appVerId:
+                description: application version id
+                type: string
+              chartAppVer:
+                description: appVersion from Chart.yaml
+                type: string
+              chartName:
+                description: The name of the chart which will be installed.
+                type: string
+              chartVersion:
+                description: Specify the exact chart version to install. If this is not specified, the latest version is installed
+                type: string
+              description:
+                description: Message got from frontend
+                type: string
+              name:
+                description: Name of the release
+                type: string
+              repoId:
+                description: id of  the repo
+                type: string
+              values:
+                description: helm release values.yaml
+                format: byte
+                type: string
+              version:
+                description: expected release version, when this version is not equal status.version, the release need upgrade this filed should be modified when any filed of the spec modified.
+                type: integer
+            required:
+            - chartName
+            - chartVersion
+            - name
+            - version
+            type: object
+          status:
+            description: HelmReleaseStatus defines the observed state of HelmRelease
+            properties:
+              deployStatus:
+                description: deploy status list of history, which will store at most 10 state
+                items:
+                  properties:
+                    deployTime:
+                      description: deploy time, upgrade time or check status time
+                      format: date-time
+                      type: string
+                    message:
+                      description: A human readable message indicating details about why the release is in this state.
+                      type: string
+                    state:
+                      description: current state of the release
+                      type: string
+                  required:
+                  - deployTime
+                  - state
+                  type: object
+                type: array
+              lastDeployed:
+                description: last deploy time or upgrade time
+                format: date-time
+                type: string
+              lastUpdate:
+                description: last update time
+                format: date-time
+                type: string
+              message:
+                description: A human readable message indicating details about why the release is in this state.
+                type: string
+              state:
+                description: current state
+                type: string
+              version:
+                description: current release version
+                type: integer
+            required:
+            - state
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/application.kubesphere.io_helmrepos.yaml

@@ -0,0 +1,142 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: helmrepos.application.kubesphere.io
+spec:
+  group: application.kubesphere.io
+  names:
+    kind: HelmRepo
+    listKind: HelmRepoList
+    plural: helmrepos
+    shortNames:
+    - hrepo
+    singular: helmrepo
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.name
+      name: name
+      type: string
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: Workspace
+      type: string
+    - jsonPath: .spec.url
+      name: url
+      type: string
+    - jsonPath: .status.state
+      name: State
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: HelmRepo is the Schema for the helmrepoes API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: HelmRepoSpec defines the desired state of HelmRepo
+            properties:
+              credential:
+                description: helm repo credential
+                properties:
+                  accessKeyID:
+                    type: string
+                  caFile:
+                    description: verify certificates of HTTPS-enabled servers using this CA bundle
+                    type: string
+                  certFile:
+                    description: identify HTTPS client using this SSL certificate file
+                    type: string
+                  insecureSkipTLSVerify:
+                    description: skip tls certificate checks for the repository, default is ture
+                    type: boolean
+                  keyFile:
+                    description: identify HTTPS client using this SSL key file
+                    type: string
+                  password:
+                    description: chart repository password
+                    type: string
+                  secretAccessKey:
+                    type: string
+                  username:
+                    description: chart repository username
+                    type: string
+                type: object
+              description:
+                description: chart repo description from frontend
+                type: string
+              name:
+                description: name of the repo
+                type: string
+              syncPeriod:
+                description: sync period in seconds, no sync when SyncPeriod=0, the minimum SyncPeriod is 180s
+                type: integer
+              url:
+                description: helm repo url
+                type: string
+              version:
+                description: expected repo version, when this version is not equal status.version, the repo need upgrade this filed should be modified when any filed of the spec modified.
+                type: integer
+            required:
+            - name
+            - url
+            type: object
+          status:
+            description: HelmRepoStatus defines the observed state of HelmRepo
+            properties:
+              data:
+                description: repo index
+                type: string
+              lastUpdateTime:
+                description: status last update time
+                format: date-time
+                type: string
+              state:
+                description: current state of the repo, successful, failed or syncing
+                type: string
+              syncState:
+                description: sync state list of history, which will store at most 10 state
+                items:
+                  properties:
+                    message:
+                      description: A human readable message indicating details about why the repo is in this state.
+                      type: string
+                    state:
+                      description: 'last sync state, valid state are: "failed", "success", and ""'
+                      type: string
+                    syncTime:
+                      format: date-time
+                      type: string
+                  required:
+                  - syncTime
+                  type: object
+                type: array
+              version:
+                description: if status.version!=spec.Version, we need sync the repo now
+                type: integer
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/cluster.kubesphere.io_clusters.yaml

@@ -0,0 +1,146 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: clusters.cluster.kubesphere.io
+spec:
+  group: cluster.kubesphere.io
+  names:
+    kind: Cluster
+    listKind: ClusterList
+    plural: clusters
+    singular: cluster
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.joinFederation
+      name: Federated
+      type: boolean
+    - jsonPath: .spec.provider
+      name: Provider
+      type: string
+    - jsonPath: .spec.enable
+      name: Active
+      type: boolean
+    - jsonPath: .status.kubernetesVersion
+      name: Version
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: Cluster is the schema for the clusters API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              connection:
+                description: Connection holds info to connect to the member cluster
+                properties:
+                  kubeconfig:
+                    description: KubeConfig content used to connect to cluster api server Should provide this field explicitly if connection type is direct. Will be populated by ks-proxy if connection type is proxy.
+                    format: byte
+                    type: string
+                  kubernetesAPIEndpoint:
+                    description: 'Kubernetes API Server endpoint. Example: https://10.10.0.1:6443 Should provide this field explicitly if connection type is direct. Will be populated by ks-apiserver if connection type is proxy.'
+                    type: string
+                  kubernetesAPIServerPort:
+                    description: KubeAPIServerPort is the port which listens for forwarding kube-apiserver traffic Only applicable when connection type is proxy.
+                    type: integer
+                  kubesphereAPIEndpoint:
+                    description: 'KubeSphere API Server endpoint. Example: http://10.10.0.11:8080 Should provide this field explicitly if connection type is direct. Will be populated by ks-apiserver if connection type is proxy.'
+                    type: string
+                  kubesphereAPIServerPort:
+                    description: KubeSphereAPIServerPort is the port which listens for forwarding kubesphere apigateway traffic Only applicable when connection type is proxy.
+                    type: integer
+                  token:
+                    description: Token used by agents of member cluster to connect to host cluster proxy. This field is populated by apiserver only if connection type is proxy.
+                    type: string
+                  type:
+                    description: type defines how host cluster will connect to host cluster ConnectionTypeDirect means direct connection, this requires   kubeconfig and kubesphere apiserver endpoint provided ConnectionTypeProxy means using kubesphere proxy, no kubeconfig   or kubesphere apiserver endpoint required
+                    type: string
+                type: object
+              enable:
+                description: Desired state of the cluster
+                type: boolean
+              joinFederation:
+                description: Join cluster as a kubefed cluster
+                type: boolean
+              provider:
+                description: Provider of the cluster, this field is just for description
+                type: string
+            type: object
+          status:
+            properties:
+              conditions:
+                description: Represents the latest available observations of a cluster's current state.
+                items:
+                  properties:
+                    lastTransitionTime:
+                      description: Last time the condition transitioned from one status to another.
+                      format: date-time
+                      type: string
+                    lastUpdateTime:
+                      description: The last time this condition was updated.
+                      format: date-time
+                      type: string
+                    message:
+                      description: A human readable message indicating details about the transition.
+                      type: string
+                    reason:
+                      description: The reason for the condition's last transition.
+                      type: string
+                    status:
+                      description: Status of the condition, one of True, False, Unknown.
+                      type: string
+                    type:
+                      description: Type of the condition
+                      type: string
+                  required:
+                  - status
+                  - type
+                  type: object
+                type: array
+              configz:
+                additionalProperties:
+                  type: boolean
+                description: Configz is status of components enabled in the member cluster. This is synchronized with member cluster every amount of time, like 5 minutes.
+                type: object
+              kubeSphereVersion:
+                description: GitVersion of the /kapis/version api response, this field is populated by cluster controller
+                type: string
+              kubernetesVersion:
+                description: GitVersion of the kubernetes cluster, this field is populated by cluster controller
+                type: string
+              nodeCount:
+                description: Count of the kubernetes cluster nodes This field may not reflect the instant status of the cluster.
+                type: integer
+              region:
+                description: Region is the name of the region in which all of the nodes in the cluster exist.  e.g. 'us-east1'.
+                type: string
+              zones:
+                description: Zones are the names of availability zones in which the nodes of the cluster exist, e.g. 'us-east1-a'.
+                items:
+                  type: string
+                type: array
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_devopsprojects.yaml

@@ -0,0 +1,52 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: devopsprojects.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    categories:
+    - devops
+    kind: DevOpsProject
+    listKind: DevOpsProjectList
+    plural: devopsprojects
+    singular: devopsproject
+  scope: Cluster
+  versions:
+  - name: v1alpha3
+    schema:
+      openAPIV3Schema:
+        description: DevOpsProject is the Schema for the devopsprojects API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: DevOpsProjectSpec defines the desired state of DevOpsProject
+            type: object
+          status:
+            description: DevOpsProjectStatus defines the observed state of DevOpsProject
+            properties:
+              adminNamespace:
+                description: 'INSERT ADDITIONAL STATUS FIELD - define observed state of cluster Important: Run "make" to regenerate code after modifying this file'
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_pipelines.yaml

@@ -0,0 +1,300 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: pipelines.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    kind: Pipeline
+    listKind: PipelineList
+    plural: pipelines
+    singular: pipeline
+  scope: Namespaced
+  versions:
+  - name: v1alpha3
+    schema:
+      openAPIV3Schema:
+        description: Pipeline is the Schema for the pipelines API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: PipelineSpec defines the desired state of Pipeline
+            properties:
+              multi_branch_pipeline:
+                properties:
+                  bitbucket_server_source:
+                    properties:
+                      api_uri:
+                        type: string
+                      credential_id:
+                        type: string
+                      discover_branches:
+                        type: integer
+                      discover_pr_from_forks:
+                        properties:
+                          strategy:
+                            type: integer
+                          trust:
+                            type: integer
+                        type: object
+                      discover_pr_from_origin:
+                        type: integer
+                      discover_tags:
+                        type: boolean
+                      git_clone_option:
+                        properties:
+                          depth:
+                            type: integer
+                          shallow:
+                            type: boolean
+                          timeout:
+                            type: integer
+                        type: object
+                      owner:
+                        type: string
+                      regex_filter:
+                        type: string
+                      repo:
+                        type: string
+                      scm_id:
+                        type: string
+                    type: object
+                  description:
+                    type: string
+                  discarder:
+                    properties:
+                      days_to_keep:
+                        type: string
+                      num_to_keep:
+                        type: string
+                    type: object
+                  git_source:
+                    properties:
+                      credential_id:
+                        type: string
+                      discover_branches:
+                        type: boolean
+                      discover_tags:
+                        type: boolean
+                      git_clone_option:
+                        properties:
+                          depth:
+                            type: integer
+                          shallow:
+                            type: boolean
+                          timeout:
+                            type: integer
+                        type: object
+                      regex_filter:
+                        type: string
+                      scm_id:
+                        type: string
+                      url:
+                        type: string
+                    type: object
+                  github_source:
+                    description: GithubSource and BitbucketServerSource have the same structure, but we don't use one due to crd errors
+                    properties:
+                      api_uri:
+                        type: string
+                      credential_id:
+                        type: string
+                      discover_branches:
+                        type: integer
+                      discover_pr_from_forks:
+                        properties:
+                          strategy:
+                            type: integer
+                          trust:
+                            type: integer
+                        type: object
+                      discover_pr_from_origin:
+                        type: integer
+                      discover_tags:
+                        type: boolean
+                      git_clone_option:
+                        properties:
+                          depth:
+                            type: integer
+                          shallow:
+                            type: boolean
+                          timeout:
+                            type: integer
+                        type: object
+                      owner:
+                        type: string
+                      regex_filter:
+                        type: string
+                      repo:
+                        type: string
+                      scm_id:
+                        type: string
+                    type: object
+                  gitlab_source:
+                    properties:
+                      api_uri:
+                        type: string
+                      credential_id:
+                        type: string
+                      discover_branches:
+                        type: integer
+                      discover_pr_from_forks:
+                        properties:
+                          strategy:
+                            type: integer
+                          trust:
+                            type: integer
+                        type: object
+                      discover_pr_from_origin:
+                        type: integer
+                      discover_tags:
+                        type: boolean
+                      git_clone_option:
+                        properties:
+                          depth:
+                            type: integer
+                          shallow:
+                            type: boolean
+                          timeout:
+                            type: integer
+                        type: object
+                      owner:
+                        type: string
+                      regex_filter:
+                        type: string
+                      repo:
+                        type: string
+                      scm_id:
+                        type: string
+                      server_name:
+                        type: string
+                    type: object
+                  multibranch_job_trigger:
+                    properties:
+                      create_action_job_to_trigger:
+                        type: string
+                      delete_action_job_to_trigger:
+                        type: string
+                    type: object
+                  name:
+                    type: string
+                  script_path:
+                    type: string
+                  single_svn_source:
+                    properties:
+                      credential_id:
+                        type: string
+                      remote:
+                        type: string
+                      scm_id:
+                        type: string
+                    type: object
+                  source_type:
+                    type: string
+                  svn_source:
+                    properties:
+                      credential_id:
+                        type: string
+                      excludes:
+                        type: string
+                      includes:
+                        type: string
+                      remote:
+                        type: string
+                      scm_id:
+                        type: string
+                    type: object
+                  timer_trigger:
+                    properties:
+                      cron:
+                        description: user in no scm job
+                        type: string
+                      interval:
+                        description: use in multi-branch job
+                        type: string
+                    type: object
+                required:
+                - name
+                - script_path
+                - source_type
+                type: object
+              pipeline:
+                properties:
+                  description:
+                    type: string
+                  disable_concurrent:
+                    type: boolean
+                  discarder:
+                    properties:
+                      days_to_keep:
+                        type: string
+                      num_to_keep:
+                        type: string
+                    type: object
+                  jenkinsfile:
+                    type: string
+                  name:
+                    type: string
+                  parameters:
+                    items:
+                      properties:
+                        default_value:
+                          type: string
+                        description:
+                          type: string
+                        name:
+                          type: string
+                        type:
+                          type: string
+                      required:
+                      - name
+                      - type
+                      type: object
+                    type: array
+                  remote_trigger:
+                    properties:
+                      token:
+                        type: string
+                    type: object
+                  timer_trigger:
+                    properties:
+                      cron:
+                        description: user in no scm job
+                        type: string
+                      interval:
+                        description: use in multi-branch job
+                        type: string
+                    type: object
+                required:
+                - name
+                type: object
+              type:
+                description: 'INSERT ADDITIONAL SPEC FIELDS - desired state of cluster Important: Run "make" to regenerate code after modifying this file'
+                type: string
+            required:
+            - type
+            type: object
+          status:
+            description: PipelineStatus defines the observed state of Pipeline
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_s2ibinaries.yaml

@@ -0,0 +1,81 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: s2ibinaries.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    kind: S2iBinary
+    listKind: S2iBinaryList
+    plural: s2ibinaries
+    singular: s2ibinary
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.fileName
+      name: FileName
+      type: string
+    - jsonPath: .spec.md5
+      name: MD5
+      type: string
+    - jsonPath: .spec.size
+      name: Size
+      type: string
+    - jsonPath: .status.phase
+      name: Phase
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: S2iBinary is the Schema for the s2ibinaries API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: S2iBinarySpec defines the desired state of S2iBinary
+            properties:
+              downloadURL:
+                description: DownloadURL in KubeSphere
+                type: string
+              fileName:
+                description: FileName is filename of binary
+                type: string
+              md5:
+                description: MD5 is Binary's MD5 Hash
+                type: string
+              size:
+                description: Size is the file size of file
+                type: string
+              uploadTimeStamp:
+                description: UploadTime is last upload time
+                format: date-time
+                type: string
+            type: object
+          status:
+            description: S2iBinaryStatus defines the observed state of S2iBinary
+            properties:
+              phase:
+                description: Phase is status of S2iBinary . Possible value is "Ready","UnableToDownload"
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_s2ibuilders.yaml

@@ -0,0 +1,475 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: s2ibuilders.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    kind: S2iBuilder
+    listKind: S2iBuilderList
+    plural: s2ibuilders
+    shortNames:
+    - s2ib
+    singular: s2ibuilder
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .status.runCount
+      name: RunCount
+      type: integer
+    - jsonPath: .status.lastRunState
+      name: LastRunState
+      type: string
+    - jsonPath: .status.lastRunName
+      name: LastRunName
+      type: string
+    - jsonPath: .status.lastRunStartTime
+      name: LastRunStartTime
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: S2iBuilder is the Schema for the s2ibuilders API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: S2iBuilderSpec defines the desired state of S2iBuilder
+            properties:
+              config:
+                description: 'INSERT ADDITIONAL SPEC FIELDS - desired state of cluster Important: Run "make" to regenerate code after modifying this file'
+                properties:
+                  addHost:
+                    description: AddHost Add a line to /etc/hosts for test purpose or private use in LAN. Its format is host:IP,multiple hosts can be added  by using multiple --add-host
+                    items:
+                      type: string
+                    type: array
+                  asDockerfile:
+                    description: AsDockerfile indicates the path where the Dockerfile should be written instead of building a new image.
+                    type: string
+                  assembleUser:
+                    description: AssembleUser specifies the user to run the assemble script in container
+                    type: string
+                  blockOnBuild:
+                    description: BlockOnBuild prevents s2i from performing a docker build operation if one is necessary to execute ONBUILD commands, or to layer source code into the container for images that don't have a tar binary available, if the image contains ONBUILD commands that would be executed.
+                    type: boolean
+                  branchExpression:
+                    description: Regular expressions, ignoring names that do not match the provided regular expression
+                    type: string
+                  buildVolumes:
+                    description: BuildVolumes specifies a list of volumes to mount to container running the build.
+                    items:
+                      type: string
+                    type: array
+                  builderBaseImageVersion:
+                    description: BuilderBaseImageVersion provides optional version information about the builder base image.
+                    type: string
+                  builderImage:
+                    description: BuilderImage describes which image is used for building the result images.
+                    type: string
+                  builderImageVersion:
+                    description: BuilderImageVersion provides optional version information about the builder image.
+                    type: string
+                  builderPullPolicy:
+                    description: BuilderPullPolicy specifies when to pull the builder image
+                    type: string
+                  callbackUrl:
+                    description: CallbackURL is a URL which is called upon successful build to inform about that fact.
+                    type: string
+                  cgroupLimits:
+                    description: CGroupLimits describes the cgroups limits that will be applied to any containers run by s2i.
+                    properties:
+                      cpuPeriod:
+                        format: int64
+                        type: integer
+                      cpuQuota:
+                        format: int64
+                        type: integer
+                      cpuShares:
+                        format: int64
+                        type: integer
+                      memoryLimitBytes:
+                        format: int64
+                        type: integer
+                      memorySwap:
+                        format: int64
+                        type: integer
+                      parent:
+                        type: string
+                    required:
+                    - cpuPeriod
+                    - cpuQuota
+                    - cpuShares
+                    - memoryLimitBytes
+                    - memorySwap
+                    - parent
+                    type: object
+                  contextDir:
+                    description: Specify a relative directory inside the application repository that should be used as a root directory for the application.
+                    type: string
+                  description:
+                    description: Description is a result image description label. The default is no description.
+                    type: string
+                  destination:
+                    description: Destination specifies a location where the untar operation will place its artifacts.
+                    type: string
+                  displayName:
+                    description: DisplayName is a result image display-name label. This defaults to the output image name.
+                    type: string
+                  dockerConfig:
+                    description: DockerConfig describes how to access host docker daemon.
+                    properties:
+                      caFile:
+                        description: CAFile is the certificate authority file path for a TLS connection
+                        type: string
+                      certFile:
+                        description: CertFile is the certificate file path for a TLS connection
+                        type: string
+                      endPoint:
+                        description: Endpoint is the docker network endpoint or socket
+                        type: string
+                      keyFile:
+                        description: KeyFile is the key file path for a TLS connection
+                        type: string
+                      tlsVerify:
+                        description: TLSVerify indicates if TLS peer must be verified
+                        type: boolean
+                      useTLS:
+                        description: UseTLS indicates if TLS must be used
+                        type: boolean
+                    required:
+                    - caFile
+                    - certFile
+                    - endPoint
+                    - keyFile
+                    - tlsVerify
+                    - useTLS
+                    type: object
+                  dockerNetworkMode:
+                    description: DockerNetworkMode is used to set the docker network setting to --net=container:<id> when the builder is invoked from a container.
+                    type: string
+                  dropCapabilities:
+                    description: DropCapabilities contains a list of capabilities to drop when executing containers
+                    items:
+                      type: string
+                    type: array
+                  environment:
+                    description: Environment is a map of environment variables to be passed to the image.
+                    items:
+                      description: EnvironmentSpec specifies a single environment variable.
+                      properties:
+                        name:
+                          type: string
+                        value:
+                          type: string
+                      required:
+                      - name
+                      - value
+                      type: object
+                    type: array
+                  excludeRegExp:
+                    description: ExcludeRegExp contains a string representation of the regular expression desired for deciding which files to exclude from the tar stream
+                    type: string
+                  export:
+                    description: Export Push the result image to specify image registry in tag
+                    type: boolean
+                  gitSecretRef:
+                    description: GitSecretRef is the BasicAuth Secret of Git Clone
+                    properties:
+                      name:
+                        description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?'
+                        type: string
+                    type: object
+                  hasOnBuild:
+                    description: HasOnBuild will be set to true if the builder image contains ONBUILD instructions
+                    type: boolean
+                  imageName:
+                    description: ImageName Contains the registry address and reponame, tag should set by field tag alone
+                    type: string
+                  imageScriptsUrl:
+                    description: ImageScriptsURL is the default location to find the assemble/run scripts for a builder image. This url can be a reference within the builder image if the scheme is specified as image://
+                    type: string
+                  imageWorkDir:
+                    description: ImageWorkDir is the default working directory for the builder image.
+                    type: string
+                  incremental:
+                    description: Incremental describes whether to try to perform incremental build.
+                    type: boolean
+                  incrementalAuthentication:
+                    description: IncrementalAuthentication holds the authentication information for pulling the previous image from private repositories
+                    properties:
+                      email:
+                        type: string
+                      password:
+                        type: string
+                      secretRef:
+                        description: LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
+                        properties:
+                          name:
+                            description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?'
+                            type: string
+                        type: object
+                      serverAddress:
+                        type: string
+                      username:
+                        type: string
+                    type: object
+                  incrementalFromTag:
+                    description: IncrementalFromTag sets an alternative image tag to look for existing artifacts. Tag is used by default if this is not set.
+                    type: string
+                  injections:
+                    description: Injections specifies a list source/destination folders that are injected to the container that runs assemble. All files we inject will be truncated after the assemble script finishes.
+                    items:
+                      description: VolumeSpec represents a single volume mount point.
+                      properties:
+                        destination:
+                          description: Destination is the path to mount the volume to - absolute or relative.
+                          type: string
+                        keep:
+                          description: Keep indicates if the mounted data should be kept in the final image.
+                          type: boolean
+                        source:
+                          description: Source is a reference to the volume source.
+                          type: string
+                      type: object
+                    type: array
+                  isBinaryURL:
+                    description: IsBinaryURL explain the type of SourceURL. If it is IsBinaryURL, it will download the file directly without using git.
+                    type: boolean
+                  keepSymlinks:
+                    description: KeepSymlinks indicates to copy symlinks as symlinks. Default behavior is to follow symlinks and copy files by content.
+                    type: boolean
+                  labelNamespace:
+                    description: LabelNamespace provides the namespace under which the labels will be generated.
+                    type: string
+                  labels:
+                    additionalProperties:
+                      type: string
+                    description: Labels specify labels and their values to be applied to the resulting image. Label keys must have non-zero length. The labels defined here override generated labels in case they have the same name.
+                    type: object
+                  layeredBuild:
+                    description: LayeredBuild describes if this is build which layered scripts and sources on top of BuilderImage.
+                    type: boolean
+                  nodeAffinityKey:
+                    description: The key of Node Affinity.
+                    type: string
+                  nodeAffinityValues:
+                    description: The values of Node Affinity.
+                    items:
+                      type: string
+                    type: array
+                  outputBuildResult:
+                    description: Whether output build result to status.
+                    type: boolean
+                  outputImageName:
+                    description: OutputImageName is a result image name without tag, default is latest. tag will append to ImageName in the end
+                    type: string
+                  preserveWorkingDir:
+                    description: PreserveWorkingDir describes if working directory should be left after processing.
+                    type: boolean
+                  previousImagePullPolicy:
+                    description: PreviousImagePullPolicy specifies when to pull the previously build image when doing incremental build
+                    type: string
+                  pullAuthentication:
+                    description: PullAuthentication holds the authentication information for pulling the Docker images from private repositories
+                    properties:
+                      email:
+                        type: string
+                      password:
+                        type: string
+                      secretRef:
+                        description: LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
+                        properties:
+                          name:
+                            description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?'
+                            type: string
+                        type: object
+                      serverAddress:
+                        type: string
+                      username:
+                        type: string
+                    type: object
+                  pushAuthentication:
+                    description: PullAuthentication holds the authentication information for pulling the Docker images from private repositories
+                    properties:
+                      email:
+                        type: string
+                      password:
+                        type: string
+                      secretRef:
+                        description: LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
+                        properties:
+                          name:
+                            description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?'
+                            type: string
+                        type: object
+                      serverAddress:
+                        type: string
+                      username:
+                        type: string
+                    type: object
+                  removePreviousImage:
+                    description: RemovePreviousImage describes if previous image should be removed after successful build. This applies only to incremental builds.
+                    type: boolean
+                  revisionId:
+                    description: The RevisionId is a branch name or a SHA-1 hash of every important thing about the commit
+                    type: string
+                  runImage:
+                    description: RunImage will trigger a "docker run ..." invocation of the produced image so the user can see if it operates as he would expect
+                    type: boolean
+                  runtimeArtifacts:
+                    description: RuntimeArtifacts specifies a list of source/destination pairs that will be copied from builder to a runtime image. Source can be a file or directory. Destination must be a directory. Regardless whether it is an absolute or relative path, it will be placed into image's WORKDIR. Destination also can be empty or equals to ".", in this case it just refers to a root of WORKDIR. In case it's empty, S2I will try to get this list from io.openshift.s2i.assemble-input-files label on a RuntimeImage.
+                    items:
+                      description: VolumeSpec represents a single volume mount point.
+                      properties:
+                        destination:
+                          description: Destination is the path to mount the volume to - absolute or relative.
+                          type: string
+                        keep:
+                          description: Keep indicates if the mounted data should be kept in the final image.
+                          type: boolean
+                        source:
+                          description: Source is a reference to the volume source.
+                          type: string
+                      type: object
+                    type: array
+                  runtimeAuthentication:
+                    description: RuntimeAuthentication holds the authentication information for pulling the runtime Docker images from private repositories.
+                    properties:
+                      email:
+                        type: string
+                      password:
+                        type: string
+                      secretRef:
+                        description: LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
+                        properties:
+                          name:
+                            description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?'
+                            type: string
+                        type: object
+                      serverAddress:
+                        type: string
+                      username:
+                        type: string
+                    type: object
+                  runtimeImage:
+                    description: RuntimeImage specifies the image that will be a base for resulting image and will be used for running an application. By default, BuilderImage is used for building and running, but the latter may be overridden.
+                    type: string
+                  runtimeImagePullPolicy:
+                    description: RuntimeImagePullPolicy specifies when to pull a runtime image.
+                    type: string
+                  scriptDownloadProxyConfig:
+                    description: ScriptDownloadProxyConfig optionally specifies the http and https proxy to use when downloading scripts
+                    properties:
+                      httpProxy:
+                        type: string
+                      httpsProxy:
+                        type: string
+                    type: object
+                  scriptsUrl:
+                    description: ScriptsURL is a URL describing where to fetch the S2I scripts from during build process. This url can be a reference within the builder image if the scheme is specified as image://
+                    type: string
+                  secretCode:
+                    description: SecretCode
+                    type: string
+                  securityOpt:
+                    description: SecurityOpt are passed as options to the docker containers launched by s2i.
+                    items:
+                      type: string
+                    type: array
+                  sourceUrl:
+                    description: SourceURL is  url of the codes such as https://github.com/a/b.git
+                    type: string
+                  tag:
+                    description: Tag is a result image tag name.
+                    type: string
+                  taintKey:
+                    description: The name of taint.
+                    type: string
+                  usage:
+                    description: Usage allows for properly shortcircuiting s2i logic when `s2i usage` is invoked
+                    type: boolean
+                  workingDir:
+                    description: WorkingDir describes temporary directory used for downloading sources, scripts and tar operations.
+                    type: string
+                  workingSourceDir:
+                    description: WorkingSourceDir describes the subdirectory off of WorkingDir set up during the repo download that is later used as the root for ignore processing
+                    type: string
+                required:
+                - imageName
+                - sourceUrl
+                type: object
+              fromTemplate:
+                description: FromTemplate define some inputs from user
+                properties:
+                  builderImage:
+                    description: BaseImage specify which version of this template to use
+                    type: string
+                  name:
+                    description: Name specify a template to use, so many fields in Config can left empty
+                    type: string
+                  parameters:
+                    description: Parameters must use with `template`, fill some parameters which template will use
+                    items:
+                      properties:
+                        defaultValue:
+                          type: string
+                        description:
+                          type: string
+                        key:
+                          type: string
+                        optValues:
+                          items:
+                            type: string
+                          type: array
+                        required:
+                          type: boolean
+                        type:
+                          type: string
+                        value:
+                          type: string
+                      type: object
+                    type: array
+                type: object
+            type: object
+          status:
+            description: S2iBuilderStatus defines the observed state of S2iBuilder
+            properties:
+              lastRunName:
+                description: LastRunState return the name of the newest run of this builder
+                type: string
+              lastRunStartTime:
+                description: LastRunStartTime return the startTime of the newest run of this builder
+                format: date-time
+                type: string
+              lastRunState:
+                description: LastRunState return the state of the newest run of this builder
+                type: string
+              runCount:
+                description: RunCount represent the sum of s2irun of this builder
+                type: integer
+            required:
+            - runCount
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_s2ibuildertemplates.yaml

@@ -0,0 +1,130 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: s2ibuildertemplates.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    categories:
+    - devops
+    kind: S2iBuilderTemplate
+    listKind: S2iBuilderTemplateList
+    plural: s2ibuildertemplates
+    shortNames:
+    - s2ibt
+    singular: s2ibuildertemplate
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.codeFramework
+      name: Framework
+      type: string
+    - jsonPath: .spec.defaultBaseImage
+      name: DefaultBaseImage
+      type: string
+    - jsonPath: .spec.version
+      name: Version
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: S2iBuilderTemplate is the Schema for the s2ibuildertemplates API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: S2iBuilderTemplateSpec defines the desired state of S2iBuilderTemplate
+            properties:
+              codeFramework:
+                description: CodeFramework means which language this template is designed for and which framework is using if has framework. Like Java, NodeJS etc
+                type: string
+              containerInfo:
+                description: Images are the images this template will use.
+                items:
+                  properties:
+                    buildVolumes:
+                      description: BuildVolumes specifies a list of volumes to mount to container running the build.
+                      items:
+                        type: string
+                      type: array
+                    builderImage:
+                      description: BaseImage are the images this template will use.
+                      type: string
+                    runtimeArtifacts:
+                      items:
+                        description: VolumeSpec represents a single volume mount point.
+                        properties:
+                          destination:
+                            description: Destination is the path to mount the volume to - absolute or relative.
+                            type: string
+                          keep:
+                            description: Keep indicates if the mounted data should be kept in the final image.
+                            type: boolean
+                          source:
+                            description: Source is a reference to the volume source.
+                            type: string
+                        type: object
+                      type: array
+                    runtimeImage:
+                      type: string
+                  type: object
+                type: array
+              defaultBaseImage:
+                description: DefaultBaseImage is the image that will be used by default
+                type: string
+              description:
+                description: Description illustrate the purpose of this template
+                type: string
+              environment:
+                description: Parameters is a set of environment variables to be passed to the image.
+                items:
+                  properties:
+                    defaultValue:
+                      type: string
+                    description:
+                      type: string
+                    key:
+                      type: string
+                    optValues:
+                      items:
+                        type: string
+                      type: array
+                    required:
+                      type: boolean
+                    type:
+                      type: string
+                    value:
+                      type: string
+                  type: object
+                type: array
+              iconPath:
+                description: IconPath is used for frontend display
+                type: string
+              version:
+                description: Version of template
+                type: string
+            type: object
+          status:
+            description: S2iBuilderTemplateStatus defines the observed state of S2iBuilderTemplate
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_s2iruns.yaml

@@ -0,0 +1,164 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: s2iruns.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    kind: S2iRun
+    listKind: S2iRunList
+    plural: s2iruns
+    shortNames:
+    - s2ir
+    singular: s2irun
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .status.runState
+      name: State
+      type: string
+    - jsonPath: .status.kubernetesJobName
+      name: K8sJobName
+      type: string
+    - jsonPath: .status.startTime
+      name: StartTime
+      type: date
+    - jsonPath: .status.completionTime
+      name: CompletionTime
+      type: date
+    - jsonPath: .status.s2iBuildResult.imageName
+      name: ImageName
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: S2iRun is the Schema for the s2iruns API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: S2iRunSpec defines the desired state of S2iRun
+            properties:
+              backoffLimit:
+                description: BackoffLimit limits the restart count of each s2irun. Default is 0
+                format: int32
+                type: integer
+              builderName:
+                description: BuilderName specify the name of s2ibuilder, required
+                type: string
+              newRevisionId:
+                description: NewRevisionId override the default NewRevisionId in its s2ibuilder.
+                type: string
+              newSourceURL:
+                description: NewSourceURL is used to download new binary artifacts
+                type: string
+              newTag:
+                description: NewTag override the default tag in its s2ibuilder, image name cannot be changed.
+                type: string
+              secondsAfterFinished:
+                description: SecondsAfterFinished if is set and greater than zero, and the job created by s2irun become successful or failed , the job will be auto deleted after SecondsAfterFinished
+                format: int32
+                type: integer
+            required:
+            - builderName
+            type: object
+          status:
+            description: S2iRunStatus defines the observed state of S2iRun
+            properties:
+              completionTime:
+                description: Represents time when the job was completed. It is not guaranteed to be set in happens-before order across separate operations. It is represented in RFC3339 form and is in UTC.
+                format: date-time
+                type: string
+              kubernetesJobName:
+                description: KubernetesJobName is the job name in k8s
+                type: string
+              logURL:
+                description: LogURL is uesd for external log handler to let user know where is log located in
+                type: string
+              runState:
+                description: RunState  indicates whether this job is done or failed
+                type: string
+              s2iBuildResult:
+                description: S2i build result info.
+                properties:
+                  commandPull:
+                    description: Command for pull image.
+                    type: string
+                  imageCreated:
+                    description: Image created time.
+                    type: string
+                  imageID:
+                    description: Image ID.
+                    type: string
+                  imageName:
+                    description: ImageName is the name of artifact
+                    type: string
+                  imageRepoTags:
+                    description: image tags.
+                    items:
+                      type: string
+                    type: array
+                  imageSize:
+                    description: The size in bytes of the image
+                    format: int64
+                    type: integer
+                type: object
+              s2iBuildSource:
+                description: S2i build source info.
+                properties:
+                  binaryName:
+                    description: Binary file Name
+                    type: string
+                  binarySize:
+                    description: Binary file Size
+                    format: int64
+                    type: integer
+                  builderImage:
+                    description: // BuilderImage describes which image is used for building the result images.
+                    type: string
+                  commitID:
+                    description: CommitID represents an arbitrary extended object reference in Git as SHA-1
+                    type: string
+                  committerEmail:
+                    description: CommitterEmail contains the e-mail of the committer
+                    type: string
+                  committerName:
+                    description: CommitterName contains the name of the committer
+                    type: string
+                  description:
+                    description: Description is a result image description label. The default is no description.
+                    type: string
+                  revisionId:
+                    description: The RevisionId is a branch name or a SHA-1 hash of every important thing about the commit
+                    type: string
+                  sourceUrl:
+                    description: SourceURL is  url of the codes such as https://github.com/a/b.git
+                    type: string
+                type: object
+              startTime:
+                description: StartTime represent when this run began
+                format: date-time
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/gateway.kubesphere.io_gateways.yaml

@@ -0,0 +1,95 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: gateways.gateway.kubesphere.io
+spec:
+  group: gateway.kubesphere.io
+  names:
+    kind: Gateway
+    listKind: GatewayList
+    plural: gateways
+    singular: gateway
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: Gateway is the Schema for the gateways API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: GatewaySpec defines the desired state of Gateway
+            properties:
+              controller:
+                properties:
+                  annotations:
+                    additionalProperties:
+                      type: string
+                    type: object
+                  config:
+                    additionalProperties:
+                      type: string
+                    type: object
+                  replicas:
+                    format: int32
+                    type: integer
+                  scope:
+                    properties:
+                      enabled:
+                        type: boolean
+                      namespace:
+                        type: string
+                    type: object
+                type: object
+              deployment:
+                properties:
+                  annotations:
+                    additionalProperties:
+                      type: string
+                    type: object
+                  replicas:
+                    format: int32
+                    type: integer
+                type: object
+              service:
+                properties:
+                  annotations:
+                    additionalProperties:
+                      type: string
+                    type: object
+                  type:
+                    description: Service Type string describes ingress methods for
+                      a service
+                    type: string
+                type: object
+            type: object
+          status:
+            type: object
+            x-kubernetes-preserve-unknown-fields: true
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/gateway.kubesphere.io_nginxes.yaml

@@ -0,0 +1,44 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: nginxes.gateway.kubesphere.io
+spec:
+  group: gateway.kubesphere.io
+  names:
+    kind: Nginx
+    listKind: NginxList
+    plural: nginxes
+    singular: nginx
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: Nginx is the Schema for the nginxes API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Spec defines the desired state of Nginx
+            type: object
+            x-kubernetes-preserve-unknown-fields: true
+          status:
+            description: Status defines the observed state of Nginx
+            type: object
+            x-kubernetes-preserve-unknown-fields: true
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}

config/crds/iam.kubesphere.io_globalrolebindings.yaml

@@ -0,0 +1,83 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: globalrolebindings.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: GlobalRoleBinding
+    listKind: GlobalRoleBindingList
+    plural: globalrolebindings
+    singular: globalrolebinding
+  scope: Cluster
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: GlobalRoleBinding is the Schema for the globalrolebindings API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          roleRef:
+            description: RoleRef can only reference a GlobalRole. If the RoleRef cannot be resolved, the Authorizer must return an error.
+            properties:
+              apiGroup:
+                description: APIGroup is the group for the resource being referenced
+                type: string
+              kind:
+                description: Kind is the type of resource being referenced
+                type: string
+              name:
+                description: Name is the name of resource being referenced
+                type: string
+            required:
+            - apiGroup
+            - kind
+            - name
+            type: object
+          subjects:
+            description: Subjects holds references to the objects the role applies to.
+            items:
+              description: Subject contains a reference to the object or user identities a role binding applies to.  This can either hold a direct API object reference, or a value for non-objects such as user and group names.
+              properties:
+                apiGroup:
+                  description: APIGroup holds the API group of the referenced subject. Defaults to "" for ServiceAccount subjects. Defaults to "rbac.authorization.k8s.io" for User and Group subjects.
+                  type: string
+                kind:
+                  description: Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount". If the Authorizer does not recognized the kind value, the Authorizer should report an error.
+                  type: string
+                name:
+                  description: Name of the object being referenced.
+                  type: string
+                namespace:
+                  description: Namespace of the referenced object.  If the object kind is non-namespace, such as "User" or "Group", and this value is not empty the Authorizer should report an error.
+                  type: string
+              required:
+              - kind
+              - name
+              type: object
+            type: array
+        required:
+        - roleRef
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_globalroles.yaml

@@ -0,0 +1,75 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: globalroles.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: GlobalRole
+    listKind: GlobalRoleList
+    plural: globalroles
+    singular: globalrole
+  scope: Cluster
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          rules:
+            description: Rules holds all the PolicyRules for this GlobalRole
+            items:
+              description: PolicyRule holds information that describes a policy rule, but does not contain information about who the rule applies to or which namespace the rule applies to.
+              properties:
+                apiGroups:
+                  description: APIGroups is the name of the APIGroup that contains the resources.  If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed.
+                  items:
+                    type: string
+                  type: array
+                nonResourceURLs:
+                  description: NonResourceURLs is a set of partial urls that a user should have access to.  *s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"),  but not both.
+                  items:
+                    type: string
+                  type: array
+                resourceNames:
+                  description: ResourceNames is an optional white list of names that the rule applies to.  An empty set means that everything is allowed.
+                  items:
+                    type: string
+                  type: array
+                resources:
+                  description: Resources is a list of resources this rule applies to.  ResourceAll represents all resources.
+                  items:
+                    type: string
+                  type: array
+                verbs:
+                  description: Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule.  VerbAll represents all kinds.
+                  items:
+                    type: string
+                  type: array
+              required:
+              - verbs
+              type: object
+            type: array
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_groupbindings.yaml

@@ -0,0 +1,64 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: groupbindings.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - group
+    kind: GroupBinding
+    listKind: GroupBindingList
+    plural: groupbindings
+    singular: groupbinding
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .groupRef.name
+      name: Group
+      type: string
+    - jsonPath: .users
+      name: Users
+      type: string
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: GroupBinding is the Schema for the groupbindings API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          groupRef:
+            description: GroupRef defines the desired relation of GroupBinding
+            properties:
+              apiGroup:
+                type: string
+              kind:
+                type: string
+              name:
+                type: string
+            type: object
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          users:
+            items:
+              type: string
+            type: array
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_groups.yaml

@@ -0,0 +1,53 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: groups.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - group
+    kind: Group
+    listKind: GroupList
+    plural: groups
+    singular: group
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: Workspace
+      type: string
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: Group is the Schema for the groups API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: GroupSpec defines the desired state of Group
+            type: object
+          status:
+            description: GroupStatus defines the observed state of Group
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_loginrecords.yaml

@@ -0,0 +1,90 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: loginrecords.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: LoginRecord
+    listKind: LoginRecordList
+    plural: loginrecords
+    singular: loginrecord
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.type
+      name: Type
+      type: string
+    - jsonPath: .spec.provider
+      name: Provider
+      type: string
+    - jsonPath: .spec.sourceIP
+      name: From
+      type: string
+    - jsonPath: .spec.success
+      name: Success
+      type: string
+    - jsonPath: .spec.reason
+      name: Reason
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              provider:
+                description: Provider of authentication, Ldap/Github etc.
+                type: string
+              reason:
+                description: States failed login attempt reason
+                type: string
+              sourceIP:
+                description: Source IP of client
+                type: string
+              success:
+                description: Successful login attempt or not
+                type: boolean
+              type:
+                description: Which authentication method used, BasicAuth/OAuth
+                type: string
+              userAgent:
+                description: User agent of login attempt
+                type: string
+            required:
+            - provider
+            - reason
+            - sourceIP
+            - success
+            - type
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_rolebases.yaml

@@ -0,0 +1,47 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: rolebases.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: RoleBase
+    listKind: RoleBaseList
+    plural: rolebases
+    singular: rolebase
+  scope: Cluster
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          role:
+            type: object
+            x-kubernetes-embedded-resource: true
+            x-kubernetes-preserve-unknown-fields: true
+        required:
+        - role
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_users.yaml

@@ -0,0 +1,92 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: users.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: User
+    listKind: UserList
+    plural: users
+    singular: user
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.email
+      name: Email
+      type: string
+    - jsonPath: .status.state
+      name: Status
+      type: string
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: User is the Schema for the users API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: UserSpec defines the desired state of User
+            properties:
+              description:
+                description: Description of the user.
+                type: string
+              displayName:
+                type: string
+              email:
+                description: Unique email address(https://www.ietf.org/rfc/rfc5322.txt).
+                type: string
+              groups:
+                items:
+                  type: string
+                type: array
+              lang:
+                description: The preferred written or spoken language for the user.
+                type: string
+              password:
+                description: password will be encrypted by mutating admission webhook
+                type: string
+            required:
+            - email
+            type: object
+          status:
+            description: UserStatus defines the observed state of User
+            properties:
+              lastLoginTime:
+                description: Last login attempt timestamp
+                format: date-time
+                type: string
+              lastTransitionTime:
+                format: date-time
+                type: string
+              reason:
+                type: string
+              state:
+                description: The user status
+                type: string
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_workspacerolebindings.yaml

@@ -0,0 +1,88 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: workspacerolebindings.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: WorkspaceRoleBinding
+    listKind: WorkspaceRoleBindingList
+    plural: workspacerolebindings
+    singular: workspacerolebinding
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: Workspace
+      type: string
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: WorkspaceRoleBinding is the Schema for the workspacerolebindings API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          roleRef:
+            description: RoleRef can only reference a WorkspaceRole. If the RoleRef cannot be resolved, the Authorizer must return an error.
+            properties:
+              apiGroup:
+                description: APIGroup is the group for the resource being referenced
+                type: string
+              kind:
+                description: Kind is the type of resource being referenced
+                type: string
+              name:
+                description: Name is the name of resource being referenced
+                type: string
+            required:
+            - apiGroup
+            - kind
+            - name
+            type: object
+          subjects:
+            description: Subjects holds references to the objects the role applies to.
+            items:
+              description: Subject contains a reference to the object or user identities a role binding applies to.  This can either hold a direct API object reference, or a value for non-objects such as user and group names.
+              properties:
+                apiGroup:
+                  description: APIGroup holds the API group of the referenced subject. Defaults to "" for ServiceAccount subjects. Defaults to "rbac.authorization.k8s.io" for User and Group subjects.
+                  type: string
+                kind:
+                  description: Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount". If the Authorizer does not recognized the kind value, the Authorizer should report an error.
+                  type: string
+                name:
+                  description: Name of the object being referenced.
+                  type: string
+                namespace:
+                  description: Namespace of the referenced object.  If the object kind is non-namespace, such as "User" or "Group", and this value is not empty the Authorizer should report an error.
+                  type: string
+              required:
+              - kind
+              - name
+              type: object
+            type: array
+        required:
+        - roleRef
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_workspaceroles.yaml

@@ -0,0 +1,83 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: workspaceroles.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: WorkspaceRole
+    listKind: WorkspaceRoleList
+    plural: workspaceroles
+    singular: workspacerole
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: Workspace
+      type: string
+    - jsonPath: .metadata.annotations.kubesphere\.io/alias-name
+      name: Alias
+      type: string
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          rules:
+            description: Rules holds all the PolicyRules for this WorkspaceRole
+            items:
+              description: PolicyRule holds information that describes a policy rule, but does not contain information about who the rule applies to or which namespace the rule applies to.
+              properties:
+                apiGroups:
+                  description: APIGroups is the name of the APIGroup that contains the resources.  If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed.
+                  items:
+                    type: string
+                  type: array
+                nonResourceURLs:
+                  description: NonResourceURLs is a set of partial urls that a user should have access to.  *s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"),  but not both.
+                  items:
+                    type: string
+                  type: array
+                resourceNames:
+                  description: ResourceNames is an optional white list of names that the rule applies to.  An empty set means that everything is allowed.
+                  items:
+                    type: string
+                  type: array
+                resources:
+                  description: Resources is a list of resources this rule applies to.  ResourceAll represents all resources.
+                  items:
+                    type: string
+                  type: array
+                verbs:
+                  description: Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule.  VerbAll represents all kinds.
+                  items:
+                    type: string
+                  type: array
+              required:
+              - verbs
+              type: object
+            type: array
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/network.kubesphere.io_ipamblocks.yaml

@@ -0,0 +1,76 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: ipamblocks.network.kubesphere.io
+spec:
+  group: network.kubesphere.io
+  names:
+    kind: IPAMBlock
+    listKind: IPAMBlockList
+    plural: ipamblocks
+    singular: ipamblock
+  scope: Cluster
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Specification of the IPAMBlock.
+            properties:
+              allocations:
+                items:
+                  type: integer
+                type: array
+              attributes:
+                items:
+                  properties:
+                    handle_id:
+                      type: string
+                    secondary:
+                      additionalProperties:
+                        type: string
+                      type: object
+                  type: object
+                type: array
+              cidr:
+                type: string
+              deleted:
+                type: boolean
+              id:
+                format: int32
+                type: integer
+              unallocated:
+                items:
+                  type: integer
+                type: array
+            required:
+            - allocations
+            - attributes
+            - cidr
+            - deleted
+            - id
+            - unallocated
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/network.kubesphere.io_ipamhandles.yaml

@@ -0,0 +1,55 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: ipamhandles.network.kubesphere.io
+spec:
+  group: network.kubesphere.io
+  names:
+    kind: IPAMHandle
+    listKind: IPAMHandleList
+    plural: ipamhandles
+    singular: ipamhandle
+  scope: Cluster
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Specification of the IPAMHandle.
+            properties:
+              block:
+                additionalProperties:
+                  type: integer
+                type: object
+              deleted:
+                type: boolean
+              handleID:
+                type: string
+            required:
+            - block
+            - deleted
+            - handleID
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/network.kubesphere.io_ippools.yaml

@@ -0,0 +1,130 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: ippools.network.kubesphere.io
+spec:
+  group: network.kubesphere.io
+  names:
+    kind: IPPool
+    listKind: IPPoolList
+    plural: ippools
+    singular: ippool
+  scope: Cluster
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              blockSize:
+                description: The block size to use for IP address assignments from this pool. Defaults to 26 for IPv4 and 112 for IPv6.
+                type: integer
+              cidr:
+                description: The pool CIDR.
+                type: string
+              disabled:
+                description: When disabled is true, IPAM will not assign addresses from this pool.
+                type: boolean
+              dns:
+                description: DNS contains values interesting for DNS resolvers
+                properties:
+                  domain:
+                    type: string
+                  nameservers:
+                    items:
+                      type: string
+                    type: array
+                  options:
+                    items:
+                      type: string
+                    type: array
+                  search:
+                    items:
+                      type: string
+                    type: array
+                type: object
+              gateway:
+                type: string
+              rangeEnd:
+                description: The last ip, inclusive
+                type: string
+              rangeStart:
+                description: The first ip, inclusive
+                type: string
+              routes:
+                items:
+                  properties:
+                    dst:
+                      type: string
+                    gateway:
+                      type: string
+                  type: object
+                type: array
+              type:
+                type: string
+              vlanConfig:
+                properties:
+                  master:
+                    type: string
+                  vlanId:
+                    format: int32
+                    type: integer
+                required:
+                - master
+                - vlanId
+                type: object
+            required:
+            - cidr
+            - type
+            type: object
+          status:
+            properties:
+              allocations:
+                type: integer
+              capacity:
+                type: integer
+              reserved:
+                type: integer
+              synced:
+                type: boolean
+              unallocated:
+                type: integer
+              workspaces:
+                additionalProperties:
+                  properties:
+                    allocations:
+                      type: integer
+                  required:
+                  - allocations
+                  type: object
+                type: object
+            required:
+            - allocations
+            - capacity
+            - unallocated
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/network.kubesphere.io_namespacenetworkpolicies.yaml

@@ -0,0 +1,179 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: namespacenetworkpolicies.network.kubesphere.io
+spec:
+  group: network.kubesphere.io
+  names:
+    categories:
+    - networking
+    kind: NamespaceNetworkPolicy
+    listKind: NamespaceNetworkPolicyList
+    plural: namespacenetworkpolicies
+    shortNames:
+    - nsnp
+    singular: namespacenetworkpolicy
+  scope: Namespaced
+  preserveUnknownFields: false
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: NamespaceNetworkPolicy is the Schema for the namespacenetworkpolicies API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: NamespaceNetworkPolicySpec provides the specification of a NamespaceNetworkPolicy
+            properties:
+              egress:
+                description: List of egress rules to be applied to the selected pods. Outgoing traffic is allowed if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic matches at least one egress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy limits all outgoing traffic (and serves solely to ensure that the pods it selects are isolated by default). This field is beta-level in 1.8
+                items:
+                  description: NetworkPolicyEgressRule describes a particular set of traffic that is allowed out of pods matched by a NetworkPolicySpec's podSelector. The traffic must match both ports and to. This type is beta-level in 1.8
+                  properties:
+                    ports:
+                      description: List of destination ports for outgoing traffic. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list.
+                      items:
+                        description: NetworkPolicyPort describes a port to allow traffic on
+                        properties:
+                          port:
+                            anyOf:
+                            - type: integer
+                            - type: string
+                            description: The port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers.
+                            x-kubernetes-int-or-string: true
+                          protocol:
+                            default: TCP
+                            description: The protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP.
+                            type: string
+                        type: object
+                      type: array
+                    to:
+                      description: List of destinations for outgoing traffic of pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all destinations (traffic not restricted by destination). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the to list.
+                      items:
+                        description: NetworkPolicyPeer describes a peer to allow traffic from. Only certain combinations of fields are allowed
+                        properties:
+                          ipBlock:
+                            description: IPBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be.
+                            properties:
+                              cidr:
+                                description: CIDR is a string representing the IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64"
+                                type: string
+                              except:
+                                description: Except is a slice of CIDRs that should not be included within an IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64" Except values will be rejected if they are outside the CIDR range
+                                items:
+                                  type: string
+                                type: array
+                            required:
+                            - cidr
+                            type: object
+                          namespace:
+                            properties:
+                              name:
+                                type: string
+                            required:
+                            - name
+                            type: object
+                          service:
+                            properties:
+                              name:
+                                type: string
+                              namespace:
+                                type: string
+                            required:
+                            - name
+                            - namespace
+                            type: object
+                        type: object
+                      type: array
+                  type: object
+                type: array
+              ingress:
+                description: List of ingress rules to be applied to the selected pods. Traffic is allowed to a pod if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic source is the pod's local node, OR if the traffic matches at least one ingress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy does not allow any traffic (and serves solely to ensure that the pods it selects are isolated by default)
+                items:
+                  description: NetworkPolicyIngressRule describes a particular set of traffic that is allowed to the pods matched by a NetworkPolicySpec's podSelector. The traffic must match both ports and from.
+                  properties:
+                    from:
+                      description: List of sources which should be able to access the pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all sources (traffic not restricted by source). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the from list.
+                      items:
+                        description: NetworkPolicyPeer describes a peer to allow traffic from. Only certain combinations of fields are allowed
+                        properties:
+                          ipBlock:
+                            description: IPBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be.
+                            properties:
+                              cidr:
+                                description: CIDR is a string representing the IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64"
+                                type: string
+                              except:
+                                description: Except is a slice of CIDRs that should not be included within an IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64" Except values will be rejected if they are outside the CIDR range
+                                items:
+                                  type: string
+                                type: array
+                            required:
+                            - cidr
+                            type: object
+                          namespace:
+                            properties:
+                              name:
+                                type: string
+                            required:
+                            - name
+                            type: object
+                          service:
+                            properties:
+                              name:
+                                type: string
+                              namespace:
+                                type: string
+                            required:
+                            - name
+                            - namespace
+                            type: object
+                        type: object
+                      type: array
+                    ports:
+                      description: List of ports which should be made accessible on the pods selected for this rule. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list.
+                      items:
+                        description: NetworkPolicyPort describes a port to allow traffic on
+                        properties:
+                          port:
+                            anyOf:
+                            - type: integer
+                            - type: string
+                            description: The port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers.
+                            x-kubernetes-int-or-string: true
+                          protocol:
+                            default: TCP
+                            description: The protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP.
+                            type: string
+                        type: object
+                      type: array
+                  type: object
+                type: array
+              policyTypes:
+                description: List of rule types that the NetworkPolicy relates to. Valid options are "Ingress", "Egress", or "Ingress,Egress". If this field is not specified, it will default based on the existence of Ingress or Egress rules; policies that contain an Egress section are assumed to affect Egress, and all policies (whether or not they contain an Ingress section) are assumed to affect Ingress. If you want to write an egress-only policy, you must explicitly specify policyTypes [ "Egress" ]. Likewise, if you want to write a policy that specifies that no egress is allowed, you must specify a policyTypes value that include "Egress" (since such a policy would not include an Egress section and would otherwise default to just [ "Ingress" ]). This field is beta-level in 1.8
+                items:
+                  description: Policy Type string describes the NetworkPolicy type This type is beta-level in 1.8
+                  type: string
+                type: array
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/quota.kubesphere.io_resourcequotas.yaml

@@ -0,0 +1,161 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: resourcequotas.quota.kubesphere.io
+spec:
+  group: quota.kubesphere.io
+  names:
+    categories:
+    - quota
+    kind: ResourceQuota
+    listKind: ResourceQuotaList
+    plural: resourcequotas
+    singular: resourcequota
+  scope: Cluster
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: WorkspaceResourceQuota sets aggregate quota restrictions enforced per workspace
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Spec defines the desired quota
+            properties:
+              quota:
+                description: Quota defines the desired quota
+                properties:
+                  hard:
+                    additionalProperties:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                    description: 'hard is the set of desired hard limits for each named resource. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/'
+                    type: object
+                  scopeSelector:
+                    description: scopeSelector is also a collection of filters like scopes that must match each object tracked by a quota but expressed using ScopeSelectorOperator in combination with possible values. For a resource to match, both scopes AND scopeSelector (if specified in spec), must be matched.
+                    properties:
+                      matchExpressions:
+                        description: A list of scope selector requirements by scope of the resources.
+                        items:
+                          description: A scoped-resource selector requirement is a selector that contains values, a scope name, and an operator that relates the scope name and values.
+                          properties:
+                            operator:
+                              description: Represents a scope's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist.
+                              type: string
+                            scopeName:
+                              description: The name of the scope that the selector applies to.
+                              type: string
+                            values:
+                              description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - operator
+                          - scopeName
+                          type: object
+                        type: array
+                    type: object
+                  scopes:
+                    description: A collection of filters that must match each object tracked by a quota. If not specified, the quota matches all objects.
+                    items:
+                      description: A ResourceQuotaScope defines a filter that must match each object tracked by a quota
+                      type: string
+                    type: array
+                type: object
+              selector:
+                additionalProperties:
+                  type: string
+                description: LabelSelector is used to select projects by label.
+                type: object
+            required:
+            - quota
+            - selector
+            type: object
+          status:
+            description: Status defines the actual enforced quota and its current usage
+            properties:
+              namespaces:
+                description: Namespaces slices the usage by project.
+                items:
+                  description: ResourceQuotaStatusByNamespace gives status for a particular project
+                  properties:
+                    hard:
+                      additionalProperties:
+                        anyOf:
+                        - type: integer
+                        - type: string
+                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                        x-kubernetes-int-or-string: true
+                      description: 'Hard is the set of enforced hard limits for each named resource. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/'
+                      type: object
+                    namespace:
+                      description: Namespace the project this status applies to
+                      type: string
+                    used:
+                      additionalProperties:
+                        anyOf:
+                        - type: integer
+                        - type: string
+                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                        x-kubernetes-int-or-string: true
+                      description: Used is the current observed total usage of the resource in the namespace.
+                      type: object
+                  required:
+                  - namespace
+                  type: object
+                type: array
+              total:
+                description: Total defines the actual enforced quota and its current usage across all projects
+                properties:
+                  hard:
+                    additionalProperties:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                    description: 'Hard is the set of enforced hard limits for each named resource. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/'
+                    type: object
+                  used:
+                    additionalProperties:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                    description: Used is the current observed total usage of the resource in the namespace.
+                    type: object
+                type: object
+            required:
+            - namespaces
+            - total
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/servicemesh.kubesphere.io_strategies.yaml

@@ -0,0 +1,911 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: strategies.servicemesh.kubesphere.io
+spec:
+  group: servicemesh.kubesphere.io
+  names:
+    kind: Strategy
+    listKind: StrategyList
+    plural: strategies
+    singular: strategy
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - description: type of strategy
+      jsonPath: .spec.type
+      name: Type
+      type: string
+    - description: destination hosts
+      jsonPath: .spec.template.spec.hosts
+      name: Hosts
+      type: string
+    - description: 'CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC. Populated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata'
+      jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: Strategy is the Schema for the strategies API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: StrategySpec defines the desired state of Strategy
+            properties:
+              governor:
+                description: Governor version, the version takes control of all incoming traffic label version value
+                type: string
+              principal:
+                description: Principal version, the one as reference version label version value
+                type: string
+              selector:
+                description: Label selector for virtual services.
+                properties:
+                  matchExpressions:
+                    description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                    items:
+                      description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                      properties:
+                        key:
+                          description: key is the label key that the selector applies to.
+                          type: string
+                        operator:
+                          description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                          type: string
+                        values:
+                          description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                          items:
+                            type: string
+                          type: array
+                      required:
+                      - key
+                      - operator
+                      type: object
+                    type: array
+                  matchLabels:
+                    additionalProperties:
+                      type: string
+                    description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                    type: object
+                type: object
+              strategyPolicy:
+                description: strategy policy, how the strategy will be applied by the strategy controller
+                type: string
+              template:
+                description: Template describes the virtual service that will be created.
+                properties:
+                  metadata:
+                    description: Metadata of the virtual services created from this template
+                    type: object
+                  spec:
+                    description: 'Configuration affecting label/content routing, sni routing,
+                      etc. See more details at: https://istio.io/docs/reference/config/networking/virtual-service.html'
+                    properties:
+                      exportTo:
+                        description: A list of namespaces to which this virtual service is
+                          exported.
+                        items:
+                          format: string
+                          type: string
+                        type: array
+                      gateways:
+                        description: The names of gateways and sidecars that should apply
+                          these routes.
+                        items:
+                          format: string
+                          type: string
+                        type: array
+                      hosts:
+                        description: The destination hosts to which traffic is being sent.
+                        items:
+                          format: string
+                          type: string
+                        type: array
+                      http:
+                        description: An ordered list of route rules for HTTP traffic.
+                        items:
+                          properties:
+                            corsPolicy:
+                              description: Cross-Origin Resource Sharing policy (CORS).
+                              properties:
+                                allowCredentials:
+                                  nullable: true
+                                  type: boolean
+                                allowHeaders:
+                                  items:
+                                    format: string
+                                    type: string
+                                  type: array
+                                allowMethods:
+                                  description: List of HTTP methods allowed to access the
+                                    resource.
+                                  items:
+                                    format: string
+                                    type: string
+                                  type: array
+                                allowOrigin:
+                                  description: The list of origins that are allowed to perform
+                                    CORS requests.
+                                  items:
+                                    format: string
+                                    type: string
+                                  type: array
+                                allowOrigins:
+                                  description: String patterns that match allowed origins.
+                                  items:
+                                    oneOf:
+                                      - not:
+                                          anyOf:
+                                            - required:
+                                                - exact
+                                            - required:
+                                                - prefix
+                                            - required:
+                                                - regex
+                                      - required:
+                                          - exact
+                                      - required:
+                                          - prefix
+                                      - required:
+                                          - regex
+                                    properties:
+                                      exact:
+                                        format: string
+                                        type: string
+                                      prefix:
+                                        format: string
+                                        type: string
+                                      regex:
+                                        description: RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
+                                        format: string
+                                        type: string
+                                    type: object
+                                  type: array
+                                exposeHeaders:
+                                  items:
+                                    format: string
+                                    type: string
+                                  type: array
+                                maxAge:
+                                  type: string
+                              type: object
+                            delegate:
+                              properties:
+                                name:
+                                  description: Name specifies the name of the delegate VirtualService.
+                                  format: string
+                                  type: string
+                                namespace:
+                                  description: Namespace specifies the namespace where the
+                                    delegate VirtualService resides.
+                                  format: string
+                                  type: string
+                              type: object
+                            fault:
+                              description: Fault injection policy to apply on HTTP traffic
+                                at the client side.
+                              properties:
+                                abort:
+                                  oneOf:
+                                    - not:
+                                        anyOf:
+                                          - required:
+                                              - httpStatus
+                                          - required:
+                                              - grpcStatus
+                                          - required:
+                                              - http2Error
+                                    - required:
+                                        - httpStatus
+                                    - required:
+                                        - grpcStatus
+                                    - required:
+                                        - http2Error
+                                  properties:
+                                    grpcStatus:
+                                      format: string
+                                      type: string
+                                    http2Error:
+                                      format: string
+                                      type: string
+                                    httpStatus:
+                                      description: HTTP status code to use to abort the Http
+                                        request.
+                                      format: int32
+                                      type: integer
+                                    percentage:
+                                      description: Percentage of requests to be aborted with
+                                        the error code provided.
+                                      properties:
+                                        value:
+                                          format: double
+                                          type: number
+                                      type: object
+                                  type: object
+                                delay:
+                                  oneOf:
+                                    - not:
+                                        anyOf:
+                                          - required:
+                                              - fixedDelay
+                                          - required:
+                                              - exponentialDelay
+                                    - required:
+                                        - fixedDelay
+                                    - required:
+                                        - exponentialDelay
+                                  properties:
+                                    exponentialDelay:
+                                      type: string
+                                    fixedDelay:
+                                      description: Add a fixed delay before forwarding the
+                                        request.
+                                      type: string
+                                    percent:
+                                      description: Percentage of requests on which the delay
+                                        will be injected (0-100).
+                                      format: int32
+                                      type: integer
+                                    percentage:
+                                      description: Percentage of requests on which the delay
+                                        will be injected.
+                                      properties:
+                                        value:
+                                          format: double
+                                          type: number
+                                      type: object
+                                  type: object
+                              type: object
+                            headers:
+                              properties:
+                                request:
+                                  properties:
+                                    add:
+                                      additionalProperties:
+                                        format: string
+                                        type: string
+                                      type: object
+                                    remove:
+                                      items:
+                                        format: string
+                                        type: string
+                                      type: array
+                                    set:
+                                      additionalProperties:
+                                        format: string
+                                        type: string
+                                      type: object
+                                  type: object
+                                response:
+                                  properties:
+                                    add:
+                                      additionalProperties:
+                                        format: string
+                                        type: string
+                                      type: object
+                                    remove:
+                                      items:
+                                        format: string
+                                        type: string
+                                      type: array
+                                    set:
+                                      additionalProperties:
+                                        format: string
+                                        type: string
+                                      type: object
+                                  type: object
+                              type: object
+                            match:
+                              items:
+                                properties:
+                                  authority:
+                                    oneOf:
+                                      - not:
+                                          anyOf:
+                                            - required:
+                                                - exact
+                                            - required:
+                                                - prefix
+                                            - required:
+                                                - regex
+                                      - required:
+                                          - exact
+                                      - required:
+                                          - prefix
+                                      - required:
+                                          - regex
+                                    properties:
+                                      exact:
+                                        format: string
+                                        type: string
+                                      prefix:
+                                        format: string
+                                        type: string
+                                      regex:
+                                        description: RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
+                                        format: string
+                                        type: string
+                                    type: object
+                                  gateways:
+                                    description: Names of gateways where the rule should be
+                                      applied.
+                                    items:
+                                      format: string
+                                      type: string
+                                    type: array
+                                  headers:
+                                    additionalProperties:
+                                      oneOf:
+                                        - not:
+                                            anyOf:
+                                              - required:
+                                                  - exact
+                                              - required:
+                                                  - prefix
+                                              - required:
+                                                  - regex
+                                        - required:
+                                            - exact
+                                        - required:
+                                            - prefix
+                                        - required:
+                                            - regex
+                                      properties:
+                                        exact:
+                                          format: string
+                                          type: string
+                                        prefix:
+                                          format: string
+                                          type: string
+                                        regex:
+                                          description: RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
+                                          format: string
+                                          type: string
+                                      type: object
+                                    type: object
+                                  ignoreUriCase:
+                                    description: Flag to specify whether the URI matching
+                                      should be case-insensitive.
+                                    type: boolean
+                                  method:
+                                    oneOf:
+                                      - not:
+                                          anyOf:
+                                            - required:
+                                                - exact
+                                            - required:
+                                                - prefix
+                                            - required:
+                                                - regex
+                                      - required:
+                                          - exact
+                                      - required:
+                                          - prefix
+                                      - required:
+                                          - regex
+                                    properties:
+                                      exact:
+                                        format: string
+                                        type: string
+                                      prefix:
+                                        format: string
+                                        type: string
+                                      regex:
+                                        description: RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
+                                        format: string
+                                        type: string
+                                    type: object
+                                  name:
+                                    description: The name assigned to a match.
+                                    format: string
+                                    type: string
+                                  port:
+                                    description: Specifies the ports on the host that is being
+                                      addressed.
+                                    type: integer
+                                  queryParams:
+                                    additionalProperties:
+                                      oneOf:
+                                        - not:
+                                            anyOf:
+                                              - required:
+                                                  - exact
+                                              - required:
+                                                  - prefix
+                                              - required:
+                                                  - regex
+                                        - required:
+                                            - exact
+                                        - required:
+                                            - prefix
+                                        - required:
+                                            - regex
+                                      properties:
+                                        exact:
+                                          format: string
+                                          type: string
+                                        prefix:
+                                          format: string
+                                          type: string
+                                        regex:
+                                          description: RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
+                                          format: string
+                                          type: string
+                                      type: object
+                                    description: Query parameters for matching.
+                                    type: object
+                                  scheme:
+                                    oneOf:
+                                      - not:
+                                          anyOf:
+                                            - required:
+                                                - exact
+                                            - required:
+                                                - prefix
+                                            - required:
+                                                - regex
+                                      - required:
+                                          - exact
+                                      - required:
+                                          - prefix
+                                      - required:
+                                          - regex
+                                    properties:
+                                      exact:
+                                        format: string
+                                        type: string
+                                      prefix:
+                                        format: string
+                                        type: string
+                                      regex:
+                                        description: RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
+                                        format: string
+                                        type: string
+                                    type: object
+                                  sourceLabels:
+                                    additionalProperties:
+                                      format: string
+                                      type: string
+                                    type: object
+                                  sourceNamespace:
+                                    description: Source namespace constraining the applicability
+                                      of a rule to workloads in that namespace.
+                                    format: string
+                                    type: string
+                                  uri:
+                                    oneOf:
+                                      - not:
+                                          anyOf:
+                                            - required:
+                                                - exact
+                                            - required:
+                                                - prefix
+                                            - required:
+                                                - regex
+                                      - required:
+                                          - exact
+                                      - required:
+                                          - prefix
+                                      - required:
+                                          - regex
+                                    properties:
+                                      exact:
+                                        format: string
+                                        type: string
+                                      prefix:
+                                        format: string
+                                        type: string
+                                      regex:
+                                        description: RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
+                                        format: string
+                                        type: string
+                                    type: object
+                                  withoutHeaders:
+                                    additionalProperties:
+                                      oneOf:
+                                        - not:
+                                            anyOf:
+                                              - required:
+                                                  - exact
+                                              - required:
+                                                  - prefix
+                                              - required:
+                                                  - regex
+                                        - required:
+                                            - exact
+                                        - required:
+                                            - prefix
+                                        - required:
+                                            - regex
+                                      properties:
+                                        exact:
+                                          format: string
+                                          type: string
+                                        prefix:
+                                          format: string
+                                          type: string
+                                        regex:
+                                          description: RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
+                                          format: string
+                                          type: string
+                                      type: object
+                                    description: withoutHeader has the same syntax with the
+                                      header, but has opposite meaning.
+                                    type: object
+                                type: object
+                              type: array
+                            mirror:
+                              properties:
+                                host:
+                                  description: The name of a service from the service registry.
+                                  format: string
+                                  type: string
+                                port:
+                                  description: Specifies the port on the host that is being
+                                    addressed.
+                                  properties:
+                                    number:
+                                      type: integer
+                                  type: object
+                                subset:
+                                  description: The name of a subset within the service.
+                                  format: string
+                                  type: string
+                              type: object
+                            mirror_percent:
+                              description: Percentage of the traffic to be mirrored by the
+                                `mirror` field.
+                              nullable: true
+                              type: integer
+                            mirrorPercent:
+                              description: Percentage of the traffic to be mirrored by the
+                                `mirror` field.
+                              nullable: true
+                              type: integer
+                            mirrorPercentage:
+                              description: Percentage of the traffic to be mirrored by the
+                                `mirror` field.
+                              properties:
+                                value:
+                                  format: double
+                                  type: number
+                              type: object
+                            name:
+                              description: The name assigned to the route for debugging purposes.
+                              format: string
+                              type: string
+                            redirect:
+                              description: A HTTP rule can either redirect or forward (default)
+                                traffic.
+                              properties:
+                                authority:
+                                  format: string
+                                  type: string
+                                redirectCode:
+                                  type: integer
+                                uri:
+                                  format: string
+                                  type: string
+                              type: object
+                            retries:
+                              description: Retry policy for HTTP requests.
+                              properties:
+                                attempts:
+                                  description: Number of retries to be allowed for a given
+                                    request.
+                                  format: int32
+                                  type: integer
+                                perTryTimeout:
+                                  description: Timeout per attempt for a given request, including
+                                    the initial call and any retries.
+                                  type: string
+                                retryOn:
+                                  description: Specifies the conditions under which retry
+                                    takes place.
+                                  format: string
+                                  type: string
+                                retryRemoteLocalities:
+                                  description: Flag to specify whether the retries should
+                                    retry to other localities.
+                                  nullable: true
+                                  type: boolean
+                              type: object
+                            rewrite:
+                              description: Rewrite HTTP URIs and Authority headers.
+                              properties:
+                                authority:
+                                  description: rewrite the Authority/Host header with this
+                                    value.
+                                  format: string
+                                  type: string
+                                uri:
+                                  format: string
+                                  type: string
+                              type: object
+                            route:
+                              description: A HTTP rule can either redirect or forward (default)
+                                traffic.
+                              items:
+                                properties:
+                                  destination:
+                                    properties:
+                                      host:
+                                        description: The name of a service from the service
+                                          registry.
+                                        format: string
+                                        type: string
+                                      port:
+                                        description: Specifies the port on the host that is
+                                          being addressed.
+                                        properties:
+                                          number:
+                                            type: integer
+                                        type: object
+                                      subset:
+                                        description: The name of a subset within the service.
+                                        format: string
+                                        type: string
+                                    type: object
+                                  headers:
+                                    properties:
+                                      request:
+                                        properties:
+                                          add:
+                                            additionalProperties:
+                                              format: string
+                                              type: string
+                                            type: object
+                                          remove:
+                                            items:
+                                              format: string
+                                              type: string
+                                            type: array
+                                          set:
+                                            additionalProperties:
+                                              format: string
+                                              type: string
+                                            type: object
+                                        type: object
+                                      response:
+                                        properties:
+                                          add:
+                                            additionalProperties:
+                                              format: string
+                                              type: string
+                                            type: object
+                                          remove:
+                                            items:
+                                              format: string
+                                              type: string
+                                            type: array
+                                          set:
+                                            additionalProperties:
+                                              format: string
+                                              type: string
+                                            type: object
+                                        type: object
+                                    type: object
+                                  weight:
+                                    format: int32
+                                    type: integer
+                                type: object
+                              type: array
+                            timeout:
+                              description: Timeout for HTTP requests, default is disabled.
+                              type: string
+                          type: object
+                        type: array
+                      tcp:
+                        description: An ordered list of route rules for opaque TCP traffic.
+                        items:
+                          properties:
+                            match:
+                              items:
+                                properties:
+                                  destinationSubnets:
+                                    description: IPv4 or IPv6 ip addresses of destination
+                                      with optional subnet.
+                                    items:
+                                      format: string
+                                      type: string
+                                    type: array
+                                  gateways:
+                                    description: Names of gateways where the rule should be
+                                      applied.
+                                    items:
+                                      format: string
+                                      type: string
+                                    type: array
+                                  port:
+                                    description: Specifies the port on the host that is being
+                                      addressed.
+                                    type: integer
+                                  sourceLabels:
+                                    additionalProperties:
+                                      format: string
+                                      type: string
+                                    type: object
+                                  sourceNamespace:
+                                    description: Source namespace constraining the applicability
+                                      of a rule to workloads in that namespace.
+                                    format: string
+                                    type: string
+                                  sourceSubnet:
+                                    description: IPv4 or IPv6 ip address of source with optional
+                                      subnet.
+                                    format: string
+                                    type: string
+                                type: object
+                              type: array
+                            route:
+                              description: The destination to which the connection should
+                                be forwarded to.
+                              items:
+                                properties:
+                                  destination:
+                                    properties:
+                                      host:
+                                        description: The name of a service from the service
+                                          registry.
+                                        format: string
+                                        type: string
+                                      port:
+                                        description: Specifies the port on the host that is
+                                          being addressed.
+                                        properties:
+                                          number:
+                                            type: integer
+                                        type: object
+                                      subset:
+                                        description: The name of a subset within the service.
+                                        format: string
+                                        type: string
+                                    type: object
+                                  weight:
+                                    format: int32
+                                    type: integer
+                                type: object
+                              type: array
+                          type: object
+                        type: array
+                      tls:
+                        items:
+                          properties:
+                            match:
+                              items:
+                                properties:
+                                  destinationSubnets:
+                                    description: IPv4 or IPv6 ip addresses of destination
+                                      with optional subnet.
+                                    items:
+                                      format: string
+                                      type: string
+                                    type: array
+                                  gateways:
+                                    description: Names of gateways where the rule should be
+                                      applied.
+                                    items:
+                                      format: string
+                                      type: string
+                                    type: array
+                                  port:
+                                    description: Specifies the port on the host that is being
+                                      addressed.
+                                    type: integer
+                                  sniHosts:
+                                    description: SNI (server name indicator) to match on.
+                                    items:
+                                      format: string
+                                      type: string
+                                    type: array
+                                  sourceLabels:
+                                    additionalProperties:
+                                      format: string
+                                      type: string
+                                    type: object
+                                  sourceNamespace:
+                                    description: Source namespace constraining the applicability
+                                      of a rule to workloads in that namespace.
+                                    format: string
+                                    type: string
+                                type: object
+                              type: array
+                            route:
+                              description: The destination to which the connection should
+                                be forwarded to.
+                              items:
+                                properties:
+                                  destination:
+                                    properties:
+                                      host:
+                                        description: The name of a service from the service
+                                          registry.
+                                        format: string
+                                        type: string
+                                      port:
+                                        description: Specifies the port on the host that is
+                                          being addressed.
+                                        properties:
+                                          number:
+                                            type: integer
+                                        type: object
+                                      subset:
+                                        description: The name of a subset within the service.
+                                        format: string
+                                        type: string
+                                    type: object
+                                  weight:
+                                    format: int32
+                                    type: integer
+                                type: object
+                              type: array
+                          type: object
+                        type: array
+                    type: object
+                type: object
+              type:
+                description: Strategy type
+                type: string
+            type: object
+          status:
+            description: StrategyStatus defines the observed state of Strategy
+            properties:
+              completionTime:
+                description: Represents time when the strategy was completed. It is represented in RFC3339 form and is in UTC.
+                format: date-time
+                type: string
+              conditions:
+                description: The latest available observations of an object's current state.
+                items:
+                  description: StrategyCondition describes current state of a strategy.
+                  properties:
+                    lastProbeTime:
+                      description: Last time the condition was checked.
+                      format: date-time
+                      type: string
+                    lastTransitionTime:
+                      description: Last time the condition transit from one status to another
+                      format: date-time
+                      type: string
+                    message:
+                      description: Human readable message indicating details about last transition.
+                      type: string
+                    reason:
+                      description: reason for the condition's last transition
+                      type: string
+                    status:
+                      description: Status of the condition, one of True, False, Unknown
+                      type: string
+                    type:
+                      description: Type of strategy condition, Complete or Failed.
+                      type: string
+                  type: object
+                type: array
+              startTime:
+                description: Represents time when the strategy was acknowledged by the controller. It is represented in RFC3339 form and is in UTC.
+                format: date-time
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/tenant.kubesphere.io_workspaces.yaml

@@ -0,0 +1,53 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: workspaces.tenant.kubesphere.io
+spec:
+  group: tenant.kubesphere.io
+  names:
+    categories:
+    - tenant
+    kind: Workspace
+    listKind: WorkspaceList
+    plural: workspaces
+    singular: workspace
+  scope: Cluster
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: Workspace is the Schema for the workspaces API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: WorkspaceSpec defines the desired state of Workspace
+            properties:
+              manager:
+                type: string
+              networkIsolation:
+                type: boolean
+            type: object
+          status:
+            description: WorkspaceStatus defines the observed state of Workspace
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/tenant.kubesphere.io_workspacetemplates.yaml

@@ -0,0 +1,126 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: workspacetemplates.tenant.kubesphere.io
+spec:
+  group: tenant.kubesphere.io
+  names:
+    categories:
+    - tenant
+    kind: WorkspaceTemplate
+    listKind: WorkspaceTemplateList
+    plural: workspacetemplates
+    singular: workspacetemplate
+  scope: Cluster
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: WorkspaceTemplate is the Schema for the workspacetemplates API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              overrides:
+                items:
+                  properties:
+                    clusterName:
+                      type: string
+                    clusterOverrides:
+                      items:
+                        properties:
+                          op:
+                            type: string
+                          path:
+                            type: string
+                          value:
+                            type: object
+                            x-kubernetes-preserve-unknown-fields: true
+                        required:
+                        - path
+                        type: object
+                      type: array
+                  required:
+                  - clusterName
+                  type: object
+                type: array
+              placement:
+                properties:
+                  clusterSelector:
+                    description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  clusters:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                      required:
+                      - name
+                      type: object
+                    type: array
+                type: object
+              template:
+                properties:
+                  metadata:
+                    type: object
+                  spec:
+                    description: WorkspaceSpec defines the desired state of Workspace
+                    properties:
+                      manager:
+                        type: string
+                      networkIsolation:
+                        type: boolean
+                    type: object
+                type: object
+            required:
+            - placement
+            - template
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/gateway/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

config/gateway/Chart.yaml

@@ -0,0 +1,6 @@
+apiVersion: v2
+appVersion: 1.16.0
+description: The Gateway helm Chart creates a Nginx Ingress Controller release by render a Nginx.gateway.kubesphere.io Kind. 
+name: gateway
+type: application
+version: 0.1.0

config/gateway/templates/nginx-ingress.yaml

@@ -0,0 +1,278 @@
+apiVersion: gateway.kubesphere.io/v1alpha1
+kind: Nginx
+metadata:
+  name: {{ .Release.Name }}-ingress
+spec:
+  fullnameOverride: {{ .Release.Name }}
+  controller:
+    # To rolling upgrade from old nginx ingress controller, we have to overide the name pattern
+    name: ""
+    image:
+    {{- with .Values.controller.image }}
+    {{- toYaml . | nindent 6 }}
+    {{- end }}
+
+    publishService:
+      enabled: {{ eq .Values.service.type "LoadBalancer" }}
+
+    # Will add custom configuration options to Nginx https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
+    {{- if .Values.controller.config }}
+    config: {{ toYaml .Values.controller.config | nindent 6 }}
+    {{- end }}
+
+    {{- if hasKey .Values.deployment.annotations "servicemesh.kubesphere.io/enabled" }}
+    podAnnotations:
+      sidecar.istio.io/inject: {{ get .Values.deployment.annotations "servicemesh.kubesphere.io/enabled" }}
+    {{- end }}
+
+    ## Annotations to be added to the controller config configuration configmap
+    ##
+    configAnnotations: {}
+
+    # Will add custom headers before sending traffic to backends according to https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/customization/custom-headers
+    proxySetHeaders: {}
+
+    # Will add custom headers before sending response traffic to the client according to: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#add-headers
+    addHeaders: {}
+
+    # Optionally customize the pod dnsConfig.
+    dnsConfig: {}
+
+
+    # Bare-metal considerations via the host network https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#via-the-host-network
+    # Ingress status was blank because there is no Service exposing the NGINX Ingress controller in a configuration using the host network, the default --publish-service flag used in standard cloud setups does not apply
+    reportNodeInternalIp: false
+
+    ## Election ID to use for status update
+    ##
+    electionID: ingress-controller-leader-{{ .Release.Name }}
+
+    ## Name of the ingress class to route through this controller
+    ##
+    ingressClass: nginx
+
+    # This section refers to the creation of the IngressClass resource
+    # IngressClass resources are supported since k8s >= 1.18
+    ingressClassResource:
+      enabled: false
+      default: false
+
+      # Parameters is a link to a custom resource containing additional
+      # configuration for the controller. This is optional if the controller
+      # does not require extra parameters.
+      parameters: {}
+
+    # labels to add to the pod container metadata
+    podLabels: {}
+    #  key: value
+
+
+    ## Limit the scope of the controller
+    ##
+{{- if .Values.controller.scope.enabled }}
+    scope:
+      enabled: true
+      namespace: {{ default .Release.Namespace .Values.controller.scope.namespace }}   # defaults to .Release.Namespace
+{{- end }}
+
+
+    ## Allows customization of the configmap / nginx-configmap namespace
+    ##
+    configMapNamespace: ""   # defaults to .Release.Namespace
+
+    ## Allows customization of the tcp-services-configmap
+    ##
+    tcp:
+      configMapNamespace: ""   # defaults to .Release.Namespace
+      ## Annotations to be added to the tcp config configmap
+      annotations: {}
+
+    ## Allows customization of the udp-services-configmap
+    ##
+    udp:
+      configMapNamespace: ""   # defaults to .Release.Namespace
+      ## Annotations to be added to the udp config configmap
+      annotations: {}
+
+
+    ## Additional command line arguments to pass to nginx-ingress-controller
+    ## E.g. to specify the default SSL certificate you can use
+    ## extraArgs:
+    ##   default-ssl-certificate: "<namespace>/<secret_name>"
+    extraArgs: {}
+
+    ## Additional environment variables to set
+    extraEnvs: []
+
+    kind: Deployment
+
+    ## Annotations to be added to the controller Deployment or DaemonSet
+    ##
+    {{- if .Values.deployment.annotations }}
+    annotations: {{ toYaml .Values.deployment.annotations | nindent 6 }}
+    {{- end }}
+
+    ## Labels to be added to the controller Deployment or DaemonSet
+    ##
+    labels: {}
+    #  keel.sh/policy: patch
+    #  keel.sh/trigger: poll
+
+
+    ## Node tolerations for server scheduling to nodes with taints
+    ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
+    ##
+    tolerations: []
+    #  - key: "key"
+    #    operator: "Equal|Exists"
+    #    value: "value"
+    #    effect: "NoSchedule|PreferNoSchedule|NoExecute(1.6 only)"
+
+    ## Affinity and anti-affinity
+    ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
+    ##
+    affinity:
+      podAntiAffinity:
+        preferredDuringSchedulingIgnoredDuringExecution:
+        - weight: 100
+          podAffinityTerm:
+            labelSelector:
+              matchExpressions:
+              - key: app.kubernetes.io/name
+                operator: In
+                values:
+                - ingress-nginx
+              - key: app.kubernetes.io/instance
+                operator: In
+                values:
+                - {{ .Release.Name }}-ingress
+              - key: app.kubernetes.io/component
+                operator: In
+                values:
+                - controller
+            topologyKey: kubernetes.io/hostname
+
+      # # An example of required pod anti-affinity
+      # podAntiAffinity:
+      #   requiredDuringSchedulingIgnoredDuringExecution:
+      #   - labelSelector:
+      #       matchExpressions:
+      #       - key: app.kubernetes.io/name
+      #         operator: In
+      #         values:
+      #         - ingress-nginx
+      #       - key: app.kubernetes.io/instance
+      #         operator: In
+      #         values:
+      #         - ingress-nginx
+      #       - key: app.kubernetes.io/component
+      #         operator: In
+      #         values:
+      #         - controller
+      #     topologyKey: "kubernetes.io/hostname"
+
+    ## Topology spread constraints rely on node labels to identify the topology domain(s) that each Node is in.
+    ## Ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
+    ##
+    topologySpreadConstraints: []
+      # - maxSkew: 1
+      #   topologyKey: failure-domain.beta.kubernetes.io/zone
+      #   whenUnsatisfiable: DoNotSchedule
+      #   labelSelector:
+      #     matchLabels:
+      #       app.kubernetes.io/instance: ingress-nginx-internal
+
+
+
+    replicaCount: {{.Values.deployment.replicas}}
+
+    minAvailable: 1
+
+    # Define requests resources to avoid probe issues due to CPU utilization in busy nodes
+    # ref: https://github.com/kubernetes/ingress-nginx/issues/4735#issuecomment-551204903
+    # Ideally, there should be no limits.
+    # https://engineering.indeedblog.com/blog/2019/12/cpu-throttling-regression-fix/
+    resources:
+    #  limits:
+    #    cpu: 100m
+    #    memory: 90Mi
+      requests:
+        cpu: 100m
+        memory: 90Mi
+
+    # Mutually exclusive with keda autoscaling
+    autoscaling:
+      enabled: false
+      minReplicas: 1
+      maxReplicas: 11
+      targetCPUUtilizationPercentage: 50
+      targetMemoryUtilizationPercentage: 50
+
+    ## Override NGINX template
+    customTemplate:
+      configMapName: ""
+      configMapKey: ""
+
+    service:
+      enabled: true
+
+{{- if .Values.service.annotations }}
+      annotations: {{ toYaml .Values.service.annotations | nindent 8 }}
+{{- end }}
+      labels: {}
+      # clusterIP: ""
+
+      ## List of IP addresses at which the controller services are available
+      ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
+      ##
+      externalIPs: []
+
+      # loadBalancerIP: ""
+      loadBalancerSourceRanges: []
+
+      ## Set external traffic policy to: "Local" to preserve source IP on
+      ## providers supporting it
+      ## Ref: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer
+      # externalTrafficPolicy: ""
+
+      # Must be either "None" or "ClientIP" if set. Kubernetes will default to "None".
+      # Ref: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
+      # sessionAffinity: ""
+
+
+      type: {{ .Values.service.type }}
+
+      # type: NodePort
+      # nodePorts:
+      #   http: 32080
+      #   https: 32443
+      #   tcp:
+      #     8080: 32808
+      nodePorts:
+        http: ""
+        https: ""
+        tcp: {}
+        udp: {}
+
+    admissionWebhooks:
+      enabled: false
+
+    metrics:
+      port: 10254
+      enabled: true
+
+      serviceMonitor:
+        enabled: true
+      prometheusRule:
+        enabled: false
+
+
+  ## Optional array of imagePullSecrets containing private registry credentials
+  ## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
+  imagePullSecrets: []
+  # - name: secretName
+
+
+
+
+

config/gateway/values.yaml

@@ -0,0 +1,29 @@
+# Default values for gateway.
+
+controller:
+  replicas: 1
+  annotations: {}
+  # add custom configuration options
+  config: {}
+  ## Limit the scope of the controller
+  ##
+  scope:
+    enabled: false
+    namespace: ""  # defaults to .Release.Namespace
+  image:
+    repository: kubesphere/nginx-ingress-controller
+    tag: "v0.48.1"
+    pullPolicy: IfNotPresent
+    digest: ""
+
+
+service:
+  ## annotations for Services, used for config Cloud LoadBalancer
+  annotations: {}
+  type: LoadBalancer
+
+## for nginx controller, same with controller
+deployment:
+  annotations: {}
+  replicas: 1
+  

config/ks-core/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

config/ks-core/Chart.yaml

@@ -0,0 +1,15 @@
+apiVersion: v2
+name: ks-core
+description: A Helm chart for KubeSphere Core components
+
+type: application
+
+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 0.1.0
+
+# This is the version number of the application being deployed. This version number should be
+# incremented each time you make changes to the application. Versions are not expected to
+# follow Semantic Versioning. They should reflect the version the application is using.
+appVersion: "v3.1.0"

config/ks-core/templates/_helpers.tpl

@@ -0,0 +1,75 @@
+{{/* vim: set filetype=mustache: */}}
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "ks-core.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+If release name contains chart name it will be used as a full name.
+*/}}
+{{- define "ks-core.fullname" -}}
+{{- if .Values.fullnameOverride }}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- $name := default .Chart.Name .Values.nameOverride }}
+{{- if contains $name .Release.Name }}
+{{- .Release.Name | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
+{{- end }}
+{{- end }}
+{{- end }}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "ks-core.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Common labels
+*/}}
+{{- define "ks-core.labels" -}}
+helm.sh/chart: {{ include "ks-core.chart" . }}
+{{ include "ks-core.selectorLabels" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end }}
+
+{{/*
+Selector labels
+*/}}
+{{- define "ks-core.selectorLabels" -}}
+app.kubernetes.io/name: {{ include "ks-core.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end }}
+
+{{/*
+Create the name of the service account to use
+*/}}
+{{- define "ks-core.serviceAccountName" -}}
+{{- if .Values.serviceAccount.create }}
+{{- default (include "ks-core.fullname" .) .Values.serviceAccount.name }}
+{{- else }}
+{{- default "default" .Values.serviceAccount.name }}
+{{- end }}
+{{- end }}
+
+{{/*
+Returns user's password or use default
+*/}}
+{{- define "getOrDefaultPass" }}
+{{- $pws := (lookup "iam.kubesphere.io/v1alpha2" "User" "" .Name) -}}
+{{- if $pws }}
+{{- $pws.spec.password  -}}
+{{- else -}}
+{{- .Default -}}
+{{- end -}}
+{{- end }}

config/ks-core/templates/account.yaml

@@ -0,0 +1,13 @@
+---
+apiVersion: iam.kubesphere.io/v1alpha2
+kind: User
+metadata:
+  name: admin
+  annotations:
+    iam.kubesphere.io/uninitialized: "true"
+    helm.sh/resource-policy: keep
+spec:
+  email: admin@kubesphere.io
+  password: "{{ include "getOrDefaultPass" (dict "Name" "admin" "Default" "$2a$10$zcHepmzfKPoxCVCYZr5K7ORPZZ/ySe9p/7IUb/8u./xHrnSX2LOCO") }}"
+status:
+  state: Active

config/ks-core/templates/ks-apiserver.yml

@@ -0,0 +1,116 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    app: ks-apiserver
+    tier: backend
+    version: {{ .Chart.AppVersion }}
+  name: ks-apiserver
+spec:
+  strategy:
+    rollingUpdate:
+      maxSurge: 0
+    type: RollingUpdate
+  replicas: {{ .Values.replicaCount }}
+  selector:
+    matchLabels:
+      app: ks-apiserver
+      tier: backend
+      # version: {{ .Chart.AppVersion }}
+  template:
+    metadata:
+      labels:
+        app: ks-apiserver
+        tier: backend
+        # version: {{ .Chart.AppVersion }}
+    spec:
+      {{- with .Values.imagePullSecrets }}
+      imagePullSecrets:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      containers:
+      - command:
+        - ks-apiserver
+        - --logtostderr=true
+        image: {{ .Values.image.ks_apiserver_repo }}:{{ .Values.image.ks_apiserver_tag | default .Chart.AppVersion }}
+        imagePullPolicy: {{ .Values.image.pullPolicy }}
+        name: ks-apiserver
+        ports:
+        - containerPort: 9090
+          protocol: TCP
+        resources:
+          {{- toYaml .Values.apiserver.resources | nindent 12 }}
+        volumeMounts:
+        - mountPath: /etc/kubesphere/ingress-controller
+          name: ks-router-config
+        - mountPath: /etc/kubesphere/
+          name: kubesphere-config
+        - mountPath: /etc/localtime
+          name: host-time
+          readOnly: true
+        {{- if .Values.apiserver.extraVolumeMounts }}
+          {{- toYaml .Values.apiserver.extraVolumeMounts | nindent 8 }}
+        {{- end }}
+        env:
+        {{- if .Values.env }}
+        {{- toYaml .Values.env | nindent 8 }}
+        {{- end }}
+        livenessProbe:
+          failureThreshold: 8
+          httpGet:
+            path: /kapis/version
+            port: 9090
+            scheme: HTTP
+          initialDelaySeconds: 15
+          timeoutSeconds: 15
+      serviceAccountName: {{ include "ks-core.serviceAccountName" . }}
+      {{- with .Values.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      volumes:
+      - configMap:
+          defaultMode: 420
+          name: ks-router-config
+        name: ks-router-config
+      - configMap:
+          defaultMode: 420
+          name: kubesphere-config
+        name: kubesphere-config
+      - hostPath:
+          path: /etc/localtime
+          type: ""
+        name: host-time
+      {{- if .Values.apiserver.extraVolumes }}
+        {{ toYaml .Values.apiserver.extraVolumes | nindent 6 }}
+      {{- end }}
+---
+
+apiVersion: v1
+kind: Service
+metadata:
+  annotations:
+    kubernetes.io/created-by: kubesphere.io/ks-apiserver
+  labels:
+    app: ks-apiserver
+    tier: backend
+    version: {{ .Chart.AppVersion }}
+  name: ks-apiserver
+spec:
+  ports:
+  - port: 80
+    protocol: TCP
+    targetPort: 9090
+  selector:
+    app: ks-apiserver
+    tier: backend
+    # version: {{ .Chart.AppVersion }}
+  type: ClusterIP

config/ks-core/templates/ks-console-config.yml

@@ -0,0 +1,28 @@
+
+apiVersion: v1
+data:
+  local_config.yaml: |
+    server:
+      http:
+        hostname: localhost
+        port: 8000
+        static:
+          production:
+            /public: server/public
+            /assets: dist/assets
+            /dist: dist
+      redis:
+        port: 6379
+        host: redis.kubesphere-system.svc
+      redisTimeout: 5000
+      sessionTimeout: 7200000
+    client:
+      version:
+        kubesphere: {{ .Chart.AppVersion }}
+        kubernetes: {{ .Values.kube_version }}
+        openpitrix: {{ .Chart.AppVersion }}
+      enableKubeConfig: true
+    defaultClusterName: {{ .Values.console.defaultClusterName }}
+kind: ConfigMap
+metadata:
+  name: ks-console-config

config/ks-core/templates/ks-console.yml

@@ -0,0 +1,108 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    app: ks-console
+    tier: frontend
+    version: {{ .Chart.AppVersion }}
+  name: ks-console
+spec:
+  strategy:
+    rollingUpdate:
+      maxSurge: 0
+    type: RollingUpdate
+  replicas: {{ .Values.replicaCount }}
+  selector:
+    matchLabels:
+      app: ks-console
+      tier: frontend
+  template:
+    metadata:
+      labels:
+        app: ks-console
+        tier: frontend
+    spec:   
+      containers:
+      - image: {{ .Values.image.ks_console_repo }}:{{ .Values.image.ks_console_tag | default .Chart.AppVersion }}
+        imagePullPolicy: {{ .Values.image.pullPolicy }}
+        name: ks-console
+        resources:
+          {{- toYaml .Values.console.resources | nindent 12 }}
+        volumeMounts:
+        - mountPath: /opt/kubesphere/console/server/local_config.yaml
+          name: ks-console-config
+          subPath: local_config.yaml
+        - mountPath: /opt/kubesphere/console/server/sample
+          name: sample-bookinfo
+        - mountPath: /etc/localtime
+          name: host-time
+          readOnly: true
+        {{- if .Values.console.extraVolumeMounts }}
+          {{- toYaml .Values.console.extraVolumeMounts | nindent 8 }}
+        {{- end }}
+        livenessProbe:
+          tcpSocket:
+            port: 8000
+          initialDelaySeconds: 15
+          timeoutSeconds: 15
+          periodSeconds: 10
+          successThreshold: 1
+          failureThreshold: 8
+      serviceAccount: {{ include "ks-core.serviceAccountName" . }}
+      serviceAccountName: {{ include "ks-core.serviceAccountName" . }}
+      {{- with .Values.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      volumes:
+      - configMap:
+          defaultMode: 420
+          name: ks-console-config
+          items:
+          - key: local_config.yaml
+            path: local_config.yaml
+        name: ks-console-config
+      - configMap:
+          defaultMode: 420
+          name: sample-bookinfo
+        name: sample-bookinfo
+      - hostPath:
+          path: /etc/localtime
+          type: ""
+        name: host-time
+      {{- if .Values.console.extraVolumes }}
+        {{ toYaml .Values.console.extraVolumes | nindent 6 }}
+      {{- end }}
+
+---
+
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    app: ks-console
+    tier: frontend
+    version: {{ .Chart.AppVersion }}
+  name: ks-console
+spec:
+  ports:
+  - name: nginx
+    port: 80
+    protocol: TCP
+    targetPort: 8000
+    {{- with .Values.console.port }}
+    nodePort:
+      {{- toYaml . | nindent 6 }}
+    {{- end }}    
+  selector:
+    app: ks-console
+    tier: frontend
+  type: {{ .Values.console.type }}

config/ks-core/templates/ks-controller-manager.yaml

@@ -0,0 +1,117 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    app: ks-controller-manager
+    tier: backend
+    version: {{ .Chart.AppVersion }}
+  name: ks-controller-manager
+spec:
+  strategy:
+    rollingUpdate:
+      maxSurge: 0
+    type: RollingUpdate
+  progressDeadlineSeconds: 600
+  replicas: {{ .Values.replicaCount }}
+  revisionHistoryLimit: 10
+  selector:
+    matchLabels:
+      app: ks-controller-manager
+      tier: backend
+      # version: {{ .Chart.AppVersion }}
+  template:
+    metadata:
+      labels:
+        app: ks-controller-manager
+        tier: backend
+        # version: {{ .Chart.AppVersion }}
+    spec:
+      {{- with .Values.imagePullSecrets }}
+      imagePullSecrets:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      containers:
+      - command:
+        - controller-manager
+        - --logtostderr=true
+        - --leader-elect=true
+        image: {{ .Values.image.ks_controller_manager_repo }}:{{ .Values.image.ks_controller_manager_tag | default .Chart.AppVersion }}
+        imagePullPolicy: {{ .Values.image.pullPolicy }}
+        name: ks-controller-manager
+        ports:
+        - containerPort: 8080
+          protocol: TCP
+        - containerPort: 8443
+          protocol: TCP
+        resources:
+          {{- toYaml .Values.controller.resources | nindent 12 }}
+        volumeMounts:
+        - mountPath: /etc/kubesphere/
+          name: kubesphere-config
+        - mountPath: /tmp/k8s-webhook-server/serving-certs
+          name: webhook-secret
+        - mountPath: /etc/localtime
+          name: host-time
+          readOnly: true
+        {{- if .Values.controller.extraVolumeMounts }}
+          {{- toYaml .Values.controller.extraVolumeMounts | nindent 8 }}
+        {{- end }}
+        env:
+        {{- if .Values.env }}
+        {{- toYaml .Values.env | nindent 8 }}
+        {{- end }}
+      dnsPolicy: ClusterFirst
+      restartPolicy: Always
+      schedulerName: default-scheduler
+      serviceAccountName: {{ include "ks-core.serviceAccountName" . }}
+      terminationGracePeriodSeconds: 30
+      volumes:
+      - name: kubesphere-config
+        configMap:
+          name: kubesphere-config
+          defaultMode: 420
+      - name: webhook-secret
+        secret:
+          defaultMode: 420
+          secretName: ks-controller-manager-webhook-cert
+      - hostPath:
+          path: /etc/localtime
+          type: ""
+        name: host-time
+      {{- if .Values.controller.extraVolumes }}
+        {{ toYaml .Values.controller.extraVolumes | nindent 6 }}
+      {{- end }}
+      {{- with .Values.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+
+---
+
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    app: ks-controller-manager
+    tier: backend
+    version: {{ .Chart.AppVersion }}
+  name: ks-controller-manager
+spec:
+  ports:
+  - port: 443
+    protocol: TCP
+    targetPort: 8443
+  selector:
+    app: ks-controller-manager
+    tier: backend
+    # version: {{ .Chart.AppVersion }}
+  sessionAffinity: None
+  type: ClusterIP

config/ks-core/templates/ks-router-cm.yaml

@@ -0,0 +1,9 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ks-router-config
+data:
+  ingress-controller-svc.yaml: |+
+{{- include "ingress-controller-svc.yaml" . }}
+  ingress-controller.yaml: |
+{{- include "ingress-controller.yaml" . }}

config/ks-core/templates/ks-router-config.tpl

@@ -0,0 +1,96 @@
+{{/* vim: set filetype=mustache: */}}
+
+{{- define "ingress-controller.yaml" }}
+    apiVersion: apps/v1
+    kind: Deployment
+    metadata:
+      name: ks-router
+    spec:
+      replicas: 1
+      selector:
+        matchLabels:
+          app: kubesphere
+          component: ks-router
+          tier: backend
+      template:
+        metadata:
+          labels:
+            app: kubesphere
+            component: ks-router
+            tier: backend
+          annotations:
+            prometheus.io/port: '10254'
+            prometheus.io/scrape: 'true'
+        spec:
+          serviceAccountName: kubesphere-router-serviceaccount
+          containers:
+            - name: nginx-ingress-controller
+              image: {{ .Values.image.nginx_ingress_controller_repo }}:{{ .Values.image.nginx_ingress_controller_tag | default .Chart.AppVersion}}
+              args:
+                - /nginx-ingress-controller
+                - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
+                - --annotations-prefix=nginx.ingress.kubernetes.io
+                - --update-status
+                - --update-status-on-shutdown
+              env:
+                - name: POD_NAME
+                  valueFrom:
+                    fieldRef:
+                      fieldPath: metadata.name
+                - name: POD_NAMESPACE
+                  valueFrom:
+                    fieldRef:
+                      fieldPath: metadata.namespace
+              ports:
+              - name: http
+                containerPort: 80
+              - name: https
+                containerPort: 443
+              livenessProbe:
+                failureThreshold: 3
+                httpGet:
+                  path: /healthz
+                  port: 10254
+                  scheme: HTTP
+                initialDelaySeconds: 10
+                periodSeconds: 10
+                successThreshold: 1
+                timeoutSeconds: 1
+              readinessProbe:
+                failureThreshold: 3
+                httpGet:
+                  path: /healthz
+                  port: 10254
+                  scheme: HTTP
+                periodSeconds: 10
+                successThreshold: 1
+                timeoutSeconds: 1
+              securityContext:
+                runAsNonRoot: false
+{{- end }}
+
+{{- define "ingress-controller-svc.yaml" }}
+    apiVersion: v1
+    kind: Service
+    metadata:
+      name: kubesphere-router-gateway
+      labels:
+        app: kubesphere
+        component: ks-router
+        tier: backend
+    spec:
+      selector:
+        app: kubesphere
+        component: ks-router
+        tier: backend
+      type: LoadBalancer
+      ports:
+        - name: http
+          protocol: TCP
+          port: 80
+          targetPort: 80
+        - name: https
+          protocol: TCP
+          port: 443
+          targetPort: 443
+{{- end }}

config/ks-core/templates/kubesphere-config.yaml

@@ -0,0 +1,34 @@
+{{- if .Values.config.create -}}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: kubesphere-config
+data:
+  kubesphere.yaml: |
+    authentication:
+      authenticateRateLimiterMaxTries: {{ .Values.config.authentication.authenticateRateLimiterMaxTries | default 10 }}
+      authenticateRateLimiterDuration: {{ .Values.config.authentication.authenticationRateLimiterDuration | default "10m0s" }}
+      loginHistoryRetentionPeriod: {{ .Values.config.authentication.loginHistoryRetentionPeriod | default "168h"  }}
+      maximumClockSkew: {{ .Values.config.authentication.maximumClockSkew | default "10s" }}
+      multipleLogin: {{ .Values.console.enableMultiLogin | default true }}
+      kubectlImage: {{ .Values.image.ks_kubectl_repo }}:{{ .Values.image.ks_kubectl_tag | default "latest" }}
+      jwtSecret: "{{ .Values.config.jwtSecret | default (randAlphaNum 32 ) }}"
+{{- if .Values.config.authentication.oauthOptions }}
+      {{- with .Values.config.authentication.oauthOptions }}
+      oauthOptions:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+{{- else if eq (default .Values.config.multicluster.clusterRole "none") "member" }}
+      oauthOptions:
+        accessTokenMaxAge: 0
+{{- end }}
+    monitoring:
+      endpoint: {{ .Values.config.monitoring.endpoint | default "http://prometheus-operated.kubesphere-monitoring-system.svc:9090" }}
+    notification:
+      endpoint: {{ .Values.config.notification.endpoint | default "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093" }}
+    {{- with .Values.config.servicemesh }}
+    servicemesh:
+      {{- toYaml . | nindent 6 }}
+    {{- end }}
+
+{{- end }}

config/ks-core/templates/kubesphere-controls-system.yaml

@@ -0,0 +1,243 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: system:kubesphere-router-clusterrole
+  annotations:
+    kubernetes.io/created-by: kubesphere.io/ks-router
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+      - endpoints
+      - nodes
+      - pods
+      - secrets
+      - namespaces
+    verbs:
+      - list
+      - watch
+      - get
+      - update
+  - apiGroups:
+      - ""
+    resources:
+      - nodes
+    verbs:
+      - get
+  - apiGroups:
+      - ""
+    resources:
+      - services
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - "extensions"
+    resources:
+      - ingresses
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - "networking.k8s.io"
+    resources:
+      - ingresses
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+        - events
+    verbs:
+        - create
+        - patch
+  - apiGroups:
+      - "extensions"
+    resources:
+      - ingresses/status
+    verbs:
+      - update
+  - apiGroups:
+      - "networking.k8s.io"
+    resources:
+      - ingresses/status
+    verbs:
+      - update
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: system:kubesphere-router-role
+  namespace: kubesphere-controls-system
+  annotations:
+    kubernetes.io/created-by: kubesphere.io/ks-router
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+      - pods
+      - secrets
+      - namespaces
+    verbs:
+      - get
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    resourceNames:
+      # Defaults to "<election-id>-<ingress-class>"
+      # Here: "<ingress-controller-leader>-<nginx>"
+      # This has to be adapted if you change either parameter
+      # when launching the nginx-ingress-controller.
+      - "ingress-controller-leader-nginx"
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    verbs:
+      - create
+  - apiGroups:
+      - ""
+    resources:
+      - endpoints
+    verbs:
+      - get
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: kubesphere-router-serviceaccount
+  namespace: kubesphere-controls-system
+  annotations:
+    kubernetes.io/created-by: kubesphere.io/ks-router
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: system:nginx-ingress-clusterrole-nisa-binding
+  annotations:
+    kubernetes.io/created-by: kubesphere.io/ks-router
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: system:kubesphere-router-clusterrole
+subjects:
+  - kind: ServiceAccount
+    name: kubesphere-router-serviceaccount
+    namespace: kubesphere-controls-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: nginx-ingress-role-nisa-binding
+  namespace: kubesphere-controls-system
+  annotations:
+    kubernetes.io/created-by: kubesphere.io/ks-router
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: system:kubesphere-router-role
+subjects:
+  - kind: ServiceAccount
+    name: kubesphere-router-serviceaccount
+    namespace: kubesphere-controls-system
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: default-http-backend
+  namespace: kubesphere-controls-system
+  labels:
+    app: kubesphere 
+    component: kubesphere-router
+    version: express-1.0.alpha
+  annotations:
+    kubernetes.io/created-by: kubesphere.io/ks-router
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: kubesphere
+      component: kubesphere-router
+  template:
+    metadata:
+      labels:
+        app: kubesphere
+        component: kubesphere-router
+    spec:
+      terminationGracePeriodSeconds: 60
+      containers:
+      - name: default-http-backend
+        # Any image is permissible as long as:
+        # 1. It serves a 404 page at /
+        # 2. It serves 200 on a /healthz endpoint
+        image: {{ .Values.image.defaultbackend_repo }}:{{ .Values.image.defaultbackend_tag | default "latest" }}
+        livenessProbe:
+          httpGet:
+            path: /healthz
+            port: 8080
+            scheme: HTTP
+          initialDelaySeconds: 30
+          timeoutSeconds: 5
+        ports:
+        - containerPort: 8080
+        resources:
+          limits:
+            cpu: 10m
+            memory: 20Mi
+          requests:
+            cpu: 10m
+            memory: 20Mi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: default-http-backend
+  namespace: kubesphere-controls-system
+  labels:
+    app: kubesphere
+    component: kubesphere-router
+  annotations:
+    kubernetes.io/created-by: kubesphere.io/ks-router
+spec:
+  ports:
+  - port: 80
+    targetPort: 8080
+  selector:
+    app: kubesphere
+    component: kubesphere-router
+
+---
+# create a seviceaccount for kubectl pod
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: kubesphere-cluster-admin
+  namespace: kubesphere-controls-system
+  annotations:
+    kubernetes.io/created-by: kubesphere.io/kubectl
+---
+# bind kubesphere-cluster-admin sa to clusterrole cluster-admin
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: system:kubesphere-cluster-admin
+  annotations:
+    kubernetes.io/created-by: kubesphere.io/kubectl
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cluster-admin
+subjects:
+  - kind: ServiceAccount
+    name: kubesphere-cluster-admin
+    namespace: kubesphere-controls-system

config/ks-core/templates/sample-bookinfo-configmap.yaml

@@ -0,0 +1,380 @@
+apiVersion: v1
+data:
+  bookinfo.yaml: |
+    apiVersion: app.k8s.io/v1beta1
+    kind: Application
+    metadata:
+      name: bookinfo
+      namespace: servicemesh
+      labels:
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      annotations:
+        servicemesh.kubesphere.io/enabled: 'true'
+    spec:
+      selector:
+        matchLabels:
+          app.kubernetes.io/version: v1
+          app.kubernetes.io/name: bookinfo
+      addOwnerRef: true
+      descriptor:
+        icons:
+          - src: '/assets/bookinfo.svg'
+      componentKinds:
+        - group: ''
+          kind: Service
+        - group: apps
+          kind: Deployment
+        - group: apps
+          kind: StatefulSet
+        - group: extensions
+          kind: Ingress
+        - group: servicemesh.kubesphere.io
+          kind: Strategy
+        - group: servicemesh.kubesphere.io
+          kind: ServicePolicy
+
+    ---
+    apiVersion: apps/v1
+    kind: Deployment
+    metadata:
+      namespace: servicemesh
+      annotations:
+        kubesphere.io/isElasticReplicas: 'false'
+        servicemesh.kubesphere.io/enabled: 'true'
+      labels:
+        app: productpage
+        version: v1
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      name: productpage-v1
+    spec:
+      replicas: 1
+      selector:
+        matchLabels:
+          app: productpage
+          version: v1
+          app.kubernetes.io/version: v1
+          app.kubernetes.io/name: bookinfo
+      template:
+        metadata:
+          labels:
+            app: productpage
+            version: v1
+            app.kubernetes.io/version: v1
+            app.kubernetes.io/name: bookinfo
+          annotations:
+            sidecar.istio.io/inject: 'true'
+        spec:
+          containers:
+            - name: productpage
+              resources:
+                requests:
+                  cpu: 10m
+                  memory: 10Mi
+                limits:
+                  cpu: '1'
+                  memory: 1000Mi
+              imagePullPolicy: IfNotPresent
+              image: {{ .Values.image.bookinfo_productpage_v1_repo }}:{{ .Values.image.bookinfo_productpage_v1_tag }}
+              ports:
+                - name: http-web
+                  protocol: TCP
+                  containerPort: 9080
+                  servicePort: 9080
+          serviceAccount: default
+      strategy:
+        type: RollingUpdate
+        rollingUpdate:
+          maxUnavailable: 25%
+          maxSurge: 25%
+    ---
+    apiVersion: v1
+    kind: Service
+    metadata:
+      namespace: servicemesh
+      labels:
+        app: productpage
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      annotations:
+        kubesphere.io/workloadType: Deployment
+        servicemesh.kubesphere.io/enabled: 'true'
+      name: productpage
+    spec:
+      type: ClusterIP
+      sessionAffinity: None
+      selector:
+        app: productpage
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      ports:
+        - name: http-web
+          protocol: TCP
+          port: 9080
+          targetPort: 9080
+
+    ---
+    apiVersion: apps/v1
+    kind: Deployment
+    metadata:
+      namespace: servicemesh
+      annotations:
+        kubesphere.io/isElasticReplicas: 'false'
+        servicemesh.kubesphere.io/enabled: 'true'
+      labels:
+        app: reviews
+        version: v1
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      name: reviews-v1
+    spec:
+      replicas: 1
+      selector:
+        matchLabels:
+          app: reviews
+          version: v1
+          app.kubernetes.io/version: v1
+          app.kubernetes.io/name: bookinfo
+      template:
+        metadata:
+          labels:
+            app: reviews
+            version: v1
+            app.kubernetes.io/version: v1
+            app.kubernetes.io/name: bookinfo
+          annotations:
+            sidecar.istio.io/inject: 'true'
+        spec:
+          containers:
+            - name: reviews
+              resources:
+                requests:
+                  cpu: 10m
+                  memory: 10Mi
+                limits:
+                  cpu: '1'
+                  memory: 1000Mi
+              imagePullPolicy: IfNotPresent
+              image: {{ .Values.image.bookinfo_reviews_v1_repo }}:{{ .Values.image.bookinfo_reviews_v1_tag }}
+              ports:
+                - name: http-web
+                  protocol: TCP
+                  containerPort: 9080
+                  servicePort: 9080
+          serviceAccount: default
+      strategy:
+        type: RollingUpdate
+        rollingUpdate:
+          maxUnavailable: 25%
+          maxSurge: 25%
+    ---
+    apiVersion: v1
+    kind: Service
+    metadata:
+      namespace: servicemesh
+      labels:
+        app: reviews
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      annotations:
+        kubesphere.io/workloadType: Deployment
+        servicemesh.kubesphere.io/enabled: 'true'
+      name: reviews
+    spec:
+      type: ClusterIP
+      sessionAffinity: None
+      selector:
+        app: reviews
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      ports:
+        - name: http-web
+          protocol: TCP
+          port: 9080
+          targetPort: 9080
+
+    ---
+    apiVersion: apps/v1
+    kind: Deployment
+    metadata:
+      namespace: servicemesh
+      annotations:
+        kubesphere.io/isElasticReplicas: 'false'
+        servicemesh.kubesphere.io/enabled: 'true'
+      labels:
+        app: details
+        version: v1
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      name: details-v1
+    spec:
+      replicas: 1
+      selector:
+        matchLabels:
+          app: details
+          version: v1
+          app.kubernetes.io/version: v1
+          app.kubernetes.io/name: bookinfo
+      template:
+        metadata:
+          labels:
+            app: details
+            version: v1
+            app.kubernetes.io/version: v1
+            app.kubernetes.io/name: bookinfo
+          annotations:
+            sidecar.istio.io/inject: 'true'
+        spec:
+          containers:
+            - name: details
+              resources:
+                requests:
+                  cpu: 10m
+                  memory: 10Mi
+                limits:
+                  cpu: '1'
+                  memory: 1000Mi
+              imagePullPolicy: IfNotPresent
+              image: {{ .Values.image.bookinfo_details_v1_repo }}:{{ .Values.image.bookinfo_details_v1_tag }}
+              ports:
+                - name: http-web
+                  protocol: TCP
+                  containerPort: 9080
+                  servicePort: 9080
+          serviceAccount: default
+      strategy:
+        type: RollingUpdate
+        rollingUpdate:
+          maxUnavailable: 25%
+          maxSurge: 25%
+    ---
+    apiVersion: v1
+    kind: Service
+    metadata:
+      namespace: servicemesh
+      labels:
+        app: details
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      annotations:
+        kubesphere.io/workloadType: Deployment
+        servicemesh.kubesphere.io/enabled: 'true'
+      name: details
+    spec:
+      type: ClusterIP
+      sessionAffinity: None
+      selector:
+        app: details
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      ports:
+        - name: http-web
+          protocol: TCP
+          port: 9080
+          targetPort: 9080
+
+    ---
+    apiVersion: apps/v1
+    kind: Deployment
+    metadata:
+      namespace: servicemesh
+      annotations:
+        kubesphere.io/isElasticReplicas: 'false'
+        servicemesh.kubesphere.io/enabled: 'true'
+      labels:
+        app: ratings
+        version: v1
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      name: ratings-v1
+    spec:
+      replicas: 1
+      selector:
+        matchLabels:
+          app: ratings
+          version: v1
+          app.kubernetes.io/version: v1
+          app.kubernetes.io/name: bookinfo
+      template:
+        metadata:
+          labels:
+            app: ratings
+            version: v1
+            app.kubernetes.io/version: v1
+            app.kubernetes.io/name: bookinfo
+          annotations:
+            sidecar.istio.io/inject: 'true'
+        spec:
+          containers:
+            - name: ratings
+              resources:
+                requests:
+                  cpu: 10m
+                  memory: 10Mi
+                limits:
+                  cpu: '1'
+                  memory: 1000Mi
+              imagePullPolicy: IfNotPresent
+              image: {{ .Values.image.bookinfo_ratings_v1_repo }}:{{ .Values.image.bookinfo_ratings_v1_tag }}
+              ports:
+                - name: http-web
+                  protocol: TCP
+                  containerPort: 9080
+                  servicePort: 9080
+          serviceAccount: default
+      strategy:
+        type: RollingUpdate
+        rollingUpdate:
+          maxUnavailable: 25%
+          maxSurge: 25%
+    ---
+    apiVersion: v1
+    kind: Service
+    metadata:
+      namespace: servicemesh
+      labels:
+        app: ratings
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      annotations:
+        kubesphere.io/workloadType: Deployment
+        servicemesh.kubesphere.io/enabled: 'true'
+      name: ratings
+    spec:
+      type: ClusterIP
+      sessionAffinity: None
+      selector:
+        app: ratings
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      ports:
+        - name: http-web
+          protocol: TCP
+          port: 9080
+          targetPort: 9080
+
+    ---
+    apiVersion: networking.k8s.io/v1
+    kind: Ingress
+    metadata:
+      namespace: servicemesh
+      labels:
+        app.kubernetes.io/version: v1
+        app.kubernetes.io/name: bookinfo
+      name: bookinfo-ingress
+    spec:
+      rules:
+        - http:
+            paths:
+            - backend:
+                service:
+                  name: productpage
+                  port:
+                    number: 9080
+              path: /
+              pathType: ImplementationSpecific
+          host: productpage.servicemesh.139.198.121.92.nip.io
+kind: ConfigMap
+metadata:
+  name: sample-bookinfo

config/ks-core/templates/serviceaccount.yaml

@@ -0,0 +1,26 @@
+{{- if .Values.serviceAccount.create -}}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: {{ include "ks-core.serviceAccountName" . }}
+  labels:
+    {{- include "ks-core.labels" . | nindent 4 }}
+  {{- with .Values.serviceAccount.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+{{- end }}
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: {{ include "ks-core.serviceAccountName" . }}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cluster-admin
+subjects:
+  - kind: ServiceAccount
+    name: {{ include "ks-core.serviceAccountName" . }}
+    namespace: {{ .Release.Namespace }}

config/ks-core/templates/webhook.yaml

@@ -0,0 +1,122 @@
+{{- $ca := genCA "ks-controller-manager-ca" 3650 }}
+{{- $cn := printf "%s-admission-webhook" .Release.Name }}
+{{- $altName1 := printf "ks-controller-manager.%s" .Release.Namespace }}
+{{- $altName2 := printf "ks-controller-manager.%s.svc" .Release.Namespace }}
+{{- $cert := genSignedCert $cn nil (list $altName1 $altName2) 3650 $ca }}
+
+apiVersion: v1
+data:
+  ca.crt: {{ b64enc $ca.Cert | quote }}
+  tls.crt: {{ b64enc  $cert.Cert | quote }}
+  tls.key: {{ b64enc  $cert.Key | quote }}
+kind: Secret
+metadata:
+  name: ks-controller-manager-webhook-cert
+type: Opaque
+---
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingWebhookConfiguration
+metadata:
+  name: users.iam.kubesphere.io
+webhooks:
+- admissionReviewVersions:
+  - v1beta1
+  clientConfig:
+    caBundle: {{ b64enc $ca.Cert | quote }}
+    service:
+      name: ks-controller-manager
+      namespace: {{ .Release.Namespace }}
+      path: /validate-email-iam-kubesphere-io-v1alpha2
+      port: 443
+  failurePolicy: Fail
+  matchPolicy: Exact
+  name: users.iam.kubesphere.io
+  namespaceSelector:
+    matchExpressions:
+    - key: control-plane
+      operator: DoesNotExist
+  objectSelector: {}
+  rules:
+  - apiGroups:
+    - iam.kubesphere.io
+    apiVersions:
+    - v1alpha2
+    operations:
+    - CREATE
+    - UPDATE
+    resources:
+    - users
+    scope: '*'
+  sideEffects: None
+  timeoutSeconds: 30
+
+---
+
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingWebhookConfiguration
+metadata:
+  name: network.kubesphere.io
+webhooks:
+  - admissionReviewVersions:
+      - v1beta1
+    clientConfig:
+      caBundle: {{ b64enc $ca.Cert | quote }}
+      service:
+        name: ks-controller-manager
+        namespace: {{ .Release.Namespace }}
+        path: /validate-network-kubesphere-io-v1alpha1
+        port: 443
+    failurePolicy: Fail
+    matchPolicy: Exact
+    name: validating-network.kubesphere.io
+    namespaceSelector:
+      matchExpressions:
+        - key: control-plane
+          operator: DoesNotExist
+    objectSelector: {}
+    rules:
+      - apiGroups:
+          - network.kubesphere.io
+        apiVersions:
+          - v1alpha1
+        operations:
+          - CREATE
+          - UPDATE
+          - DELETE
+        resources:
+          - ippools
+        scope: '*'
+    sideEffects: None
+    timeoutSeconds: 30
+
+---
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingWebhookConfiguration
+metadata:
+  name: resourcesquotas.quota.kubesphere.io
+webhooks:
+  - admissionReviewVersions:
+      - v1beta1
+    clientConfig:
+      caBundle: {{ b64enc $ca.Cert | quote }}
+      service:
+        name: ks-controller-manager
+        namespace: {{ .Release.Namespace }}
+        path: /validate-quota-kubesphere-io-v1alpha2
+        port: 443
+    failurePolicy: Ignore
+    matchPolicy: Exact
+    name: resourcesquotas.quota.kubesphere.io
+    namespaceSelector: {}
+    objectSelector: {}
+    rules:
+      - apiGroups:
+          - '*'
+        apiVersions:
+          - '*'
+        operations:
+          - CREATE
+        resources:
+          - pods
+        scope: '*'
+    sideEffects: None

config/ks-core/values.yaml

@@ -0,0 +1,158 @@
+# Default values for ks-core.
+
+replicaCount: 1
+
+image:
+  # Overrides the image tag whose default is the chart appVersion.
+  ks_controller_manager_repo: kubesphere/ks-controller-manager
+  ks_controller_manager_tag: ""
+
+  ks_apiserver_repo: kubesphere/ks-apiserver
+  ks_apiserver_tag: ""
+  ks_console_repo: "kubesphere/ks-console"
+  ks_console_tag: ""
+
+  ks_kubectl_repo: kubesphere/kubectl
+  ks_kubectl_tag: ""
+
+  nginx_ingress_controller_repo: kubesphere/nginx-ingress-controller
+  nginx_ingress_controller_tag: "v0.35.0"
+  defaultbackend_repo: "mirrorgooglecontainers/defaultbackend-amd64"
+  defaultbackend_tag: "1.4"
+
+  bookinfo_productpage_v1_repo: kubesphere/examples-bookinfo-productpage-v1
+  bookinfo_productpage_v1_tag: "1.16.2"
+
+  bookinfo_reviews_v1_repo: kubesphere/examples-bookinfo-reviews-v1
+  bookinfo_reviews_v1_tag: "1.16.2"
+
+  bookinfo_details_v1_repo: kubesphere/examples-bookinfo-details-v1
+  bookinfo_details_v1_tag: "1.16.2"
+
+  bookinfo_ratings_v1_repo: kubesphere/examples-bookinfo-ratings-v1
+  bookinfo_ratings_v1_tag: "1.16.3"
+
+  pullPolicy: IfNotPresent
+
+
+config:
+  # Specifies whether the kubesphere-config configmap should be created
+  create: true
+  authentication: {}
+  # Jwt Secret is required by ks-apiserver, a random string would be generated if it's empty
+  jwtSecret: ""
+  multicluster: {}
+  monitoring: {}
+  notification: {}
+
+imagePullSecrets: []
+nameOverride: ""
+fullnameOverride: ""
+
+serviceAccount:
+  # Specifies whether a service account should be created
+  create: true
+  # Annotations to add to the service account
+  annotations: {}
+  # The name of the service account to use.
+  # If not set and create is true, a name is generated using the fullname template
+  name: "kubesphere"
+
+podAnnotations: {}
+
+podSecurityContext: {}
+  # fsGroup: 2000
+
+securityContext: {}
+  # capabilities:
+  #   drop:
+  #   - ALL
+  # readOnlyRootFilesystem: true
+  # runAsNonRoot: true
+  # runAsUser: 1000
+
+# Kubernetes Version shows in KubeSphere console
+kube_version: "v1.19.4"
+
+env: []
+
+tolerations: 
+  - key: node-role.kubernetes.io/master
+    effect: NoSchedule
+  - key: CriticalAddonsOnly
+    operator: Exists
+  - effect: NoExecute
+    key: node.kubernetes.io/not-ready
+    operator: Exists
+    tolerationSeconds: 60
+  - effect: NoExecute
+    key: node.kubernetes.io/unreachable
+    operator: Exists
+    tolerationSeconds: 60
+
+affinity: {}
+
+nodeSelector: {}
+
+## deployment specific configuration
+
+apiserver:
+  resources:
+    limits:
+      cpu: 1
+      memory: 1024Mi
+    requests:
+      cpu: 20m
+      memory: 100Mi
+
+  extraVolumeMounts: []
+  ## Additional volumeMounts to the apiserver container.
+  #  - name: example-config
+  #   mountPath: /etc/kubesphere/example
+
+  extraVolumes: []
+  ## Additional volumes to the apiserver pod.
+  #  - name: example-config
+  #    emptyDir: {}
+
+console:
+  port: 30880
+  type: NodePort
+  defaultClusterName: "default"
+  resources:
+    limits:
+      cpu: 1
+      memory: 1024Mi
+    requests:
+      cpu: 20m
+      memory: 100Mi
+
+  extraVolumeMounts: []
+  ## Additional volumeMounts to the console container.
+  #  - name: example-config
+  #   mountPath: /etc/kubesphere/example
+
+  extraVolumes: []
+  ## Additional volumes to the console pod.
+  #  - name: example-config
+  #    emptyDir: {}
+
+controller:
+  resources:
+    limits:
+      cpu: 1
+      memory: 1000Mi
+    requests:
+      cpu: 30m
+      memory: 50Mi
+
+  extraVolumeMounts: []
+  ## Additional volumeMounts to the controller container.
+  #  - name: example-config
+  #   mountPath: /etc/kubesphere/example
+
+  extraVolumes: []
+  ## Additional volumes to the controller pod.
+  #  - name: example-config
+  #    emptyDir: {}
+

config/watches.yaml

@@ -0,0 +1,14 @@
+# Use the 'create api' subcommand to add watches to this file.
+- group: gateway.kubesphere.io
+  version: v1alpha1
+  kind: Nginx
+  chart: /var/helm-charts/ingress-nginx
+- group: gateway.kubesphere.io
+  version: v1alpha1
+  kind: Gateway
+  chart: /var/helm-charts/gateway
+# uncomment the following 2 lines, if only allow to run global scope or namespaced scope.
+# Leave the option to CR, when allow to run global and namespaced together.
+  # overrideValues:
+  #   controller.scope.enabled: $SCOPE_ENABLED
+#+kubebuilder:scaffold:watch

docs/build-multiarch-images.md

@@ -0,0 +1,29 @@
+# build mulit-arch image with docker buildx
+
+## set up env for docker buildx
+
+run follow commands
+
+```
+docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
+docker buildx rm arm64builder
+docker buildx create --name arm64builder
+docker buildx use arm64builder
+docker buildx inspect --bootstrap
+```
+
+## build images
+
+If build multi platform one time, can not use `--load` param, You must have privileges to push images to `$REPO` 
+
+```
+docker buildx build --platform="linux/amd64,linux/arm64" . -t $REPO/ks-apiserver:$TAG -f build-multiarch/Dockerfile --target=ks-apiserver --push
+docker buildx build --platform="linux/amd64,linux/arm64" . -t $REPO/ks-controller-manager:$TAG -f build-multiarch/Dockerfile --target=ks-controller-manager --push
+```
+
+Also you can build a image for local build with only one arch
+
+```
+docker buildx build --platform="linux/arm64" . -t "ks-apiserver:arm64" -f build-multiarch/Dockerfile --target=ks-apiserver --load
+docker buildx build --platform="linux/amd64" . -t "ks-apiserver:amd64" -f build-multiarch/Dockerfile --target=ks-apiserver --load
+```

docs/code-of-conduct.md

@@ -1,14 +0,0 @@
-# KubeSphere Code of Conduct
-
-KubeSphere follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).  
-
-# Best practice of committing code
-Besides following above conduct from CNCF, we also hope every contributor in this project could help us to improve the quality of code, something you should know before checking in any new code:
- - As gopher, make sure you already read [the conduct of Go language](https://golang.org/conduct) and [the instruction of writting Go](https://golang.org/doc/effective_go.html).  
- - Fork the project under your account and make the changes you want there.  
- - Execute 'go fmt' for every piece of new code.  
- - Every pulling request(PR) would be better constructed with only one commit, this could help code reviewer to go through your code efficiently, also helpful for every follower of this project to understand what happens in this PR. If you need to make any further code change to address the comments from reviewers, which means some new commits will be generated under this PR, you need to use 'git rebase' to combine those commits together.
- - Every PR should only solve one problem or provide one feature, don't put several different fixes into one PR.  
- - At lease two code reviewers should involve into code reviewing process. 
- - Please introduce new third-party packages as little as possible to reduce the vendor dependency of this project. For example, don't import a full unit converting package but only use one function from it. For this case, you'd better write that function by yourself.
- - more.