Merge pull request #3806 from yunkunrao/master

Fix metering chart display error when switched into network flows consumption.

.dockerignore

@@ -1 +1,3 @@
-tmp/
+# exclude all files and folders except bin folder
+**
+!bin

.github/.stale.yaml

@@ -1,16 +0,0 @@
-# Number of days of inactivity before an issue becomes stale
-daysUntilStale: 30
-# Number of days of inactivity before a stale issue is closed
-daysUntilClose: 14
-# Issues with these labels will never be considered stale
-exemptLabels:
-  - frozen
-staleLabel: stale
-# Comment to post when marking an issue as stale. Set to `false` to disable
-markComment: >
-  This issue has been automatically marked as stale because it has not had
-  recent activity. It will be closed if no further activity occurs. Any further update will
-  cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
-# Comment to post when closing a stale issue. Set to `false` to disable
-closeComment: >
-  This issue is being automatically closed due to inactivity.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -3,40 +3,39 @@ name: Bug report
 about: Create a report to help us improve
 ---
 
- 
-**General remarks**
+<!--
+You don't need to remove this comment section, it's invisible on the issues page.
 
-> Please delete this section including header before submitting
-> 也可以使用中文
->
-> This form is to report bugs. For general usage questions refer to our Slack channel
->        [KubeSphere-users](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTdkNTc3OTdmNzdiODViZjViNTU5ZDY3M2I2MzY4MTI4OGZlOTJmMDg5ZTFiMDAwYzNlZDY5NjA0NzZlNDU5NmY)
+## General remarks
 
-**Describe the bug(描述下问题)**
+* Attention, please fill out this issues form using English only!
+* 注意！GitHub Issue 仅支持英文，中文 Issue 请在 [论坛](https://kubesphere.com.cn/forum/) 提交。
+* This form is to report bugs. For general usage questions you can join our Slack channel
+       [KubeSphere-users](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE)
+-->
+
+**Describe the Bug**
 A clear and concise description of what the bug is.
 
 For UI issues please also add a screenshot that shows the issue.
 
-**Versions used(KubeSphere/Kubernetes的版本)**
+**Versions Used**
 KubeSphere:
 Kubernetes: (If KubeSphere installer used, you can skip this)
 
 
-**Environment(环境的硬件配置)**
-How many nodes and their hardware configuration: 
-
-For example:
-3 masters:  8cpu/8g
-3 nodes: 8cpu/16g
+**Environment**
+How many nodes and their hardware configuration:
 
+For example: CentOS 7.5 / 3 masters:  8cpu/8g; 3 nodes: 8cpu/16g
 (and other info are welcomed to help us debugging)
 
-**To Reproduce(复现步骤)**
+**How To Reproduce**
 Steps to reproduce the behavior:
 1. Go to '...'
 2. Click on '....'
 3. Scroll down to '....'
 4. See error
 
-**Expected behavior(预期行为)**
+**Expected behavior**
 A clear and concise description of what you expected to happen.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,69 @@
+---
+name: Feature Request
+about: Have a good idea? Please don't hesitate to write it down, describe the new feature.
+---
+
+<!--
+You don't need to remove this comment section, it's invisible on the issues page.
+
+## General remarks
+
+* Attention, please fill out this issues form using English only!
+* 注意！GitHub Issue 仅支持英文，中文 Issue 请在 [论坛](https://kubesphere.com.cn/forum/) 提交。
+* This form is to report bugs. For general usage questions, you can join our Slack channel
+       [KubeSphere-users](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE)
+-->
+
+**What's it about?**
+<!--
+A clear and concise description of what this feature request is.
+-->
+
+**What's the reason why we need it?**
+<!--
+Please tell us if you think it's a necessary feature for Kubesphere. Give us as many details about it as you can. 
+Two or more use cases might be very helpful when other contributors try to go through this request. If you have some references,
+please just add it below.
+-->
+
+I believe this is an important feature for Kubesphere. There're a few use cases:
+
+* case one
+* case two
+* ...
+
+Please leave your comments below if there's anyone agrees with me. Or just give me a thumb up.
+
+**Area Suggestion**
+<!--
+In order to have a clear issue list, giving an accuracy area is necessary. If you are not sure about it, please just leave it alone.
+
+You can find some possible areas below. Please attention, sometimes crossing multiple areas might be possible. So, you 
+can keep one or more areas in this issue.
+
+> /area alerting
+> /area api
+> /area apiserver
+> /area app-management
+> /area audit
+> /area console
+> /area devops
+> /area documentation
+> /area edge
+> /area iam
+> /area installation
+> /area logging
+> /area microservice
+> /area monitoring
+> /area multicluster
+> /area networking
+> /area notification
+> /area observability
+> /area performance
+> /area security
+> /area storage
+> /area test
+> /area upgrade
+-->
+
+/kind feature-request

.github/ISSUE_TEMPLATE/installation_failure.md

@@ -1,16 +1,30 @@
 ---
-name: 安装问题
-about: Create a report to help us improve
+name: Installation Issue
+about: Create an issue to help us improve installation
 ---
 
-[备注]: <> (请补全下面信息，帮助我们更快地定位问题。提交问题前预览下issue，看下是否有格式错误)
+## English only!
 
-**问题描述**
+**注意！GitHub Issue 仅支持英文，中文 Issue 请在 [论坛](https://kubesphere.com.cn/forum/) 提交。**
 
-**安装环境的硬件配置**
+**General remarks**
 
-[备注]: <> (请说明节点的运行环境，是否是物理机，云主机，VMware虚拟机)
+> Please delete this section including header before submitting
+>
+> This form is to report installation issues. For general usage questions you can refer to [KubeSphere Documentation](https://kubesphere.io/docs) or join our Slack channel
+>        [KubeSphere-users](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE)
 
-**错误信息或截图**
+**What's your question**
 
-**Installer版本**
+
+**Environment: OS & Hardware Information**
+
+> Important: You must describe your environment clearly, e.g. VMware or Bare Metal, CentOS 7.5, 8 C / 16 G (If you install on Linux), Or Kubernetes v1.16 (If you install on K8s).
+
+
+**Error logs or message (Attach logs or screenshot)**
+
+
+**Installer Version**
+
+> e.g. v2.1.0, v2.1.1, v3.0.0

.github/stale.yml

@@ -0,0 +1,20 @@
+# Number of days of inactivity before an issue becomes stale
+daysUntilStale: 90
+# Number of days of inactivity before a stale issue is closed
+daysUntilClose: 30
+# Issues with these labels will never be considered stale
+exemptLabels:
+  - frozen
+  - kind/feature-request
+  - kind/feature
+  - kind/security
+  - kind/design
+staleLabel: stale
+# Comment to post when marking an issue as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed if no further activity occurs. Any further update will
+  cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
+# Comment to post when closing a stale issue. Set to `false` to disable
+closeComment: >
+  This issue is being automatically closed due to inactivity.

.github/workflows/build.yml

@@ -0,0 +1,72 @@
+name: Go
+
+on:
+  push:
+    branches:
+      - 'master'
+      - 'release*'
+    tags:
+      - 'v*'
+  pull_request:
+    branches:
+      - 'master'
+      - 'release*'
+
+jobs:
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    env:
+      GO111MODULE: on
+    steps:
+      - name: Set up Go 1.13
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.13
+        id: go
+
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Check pr is properly formatted
+        run: diff -u <(echo -n) <(gofmt -d ./pkg ./cmd ./tools ./test)
+
+      - name: Verify goimports
+        run: go get -u golang.org/x/tools/cmd/goimports && bash hack/verify-goimports.sh
+
+      - name: Downloading go dependencies
+        run: go mod vendor
+
+      - name: Install kubebuilder
+        run: bash hack/install_kubebuilder.sh
+
+      - name: Build
+        run: make all
+
+      - name: Make OpenAPI Spec
+        run: make openapi
+
+      - name: Uploading code coverage
+        uses: codecov/codecov-action@v1
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          file: ./coverage.txt
+          flags: unittests
+          name: codecov-umbrella
+          fail_ci_if_error: false
+
+      - name: Get branch name
+        id: extract_branch
+        shell: bash
+        run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/})"
+
+      - name: Build and push docker images
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
+        if: github.event_name == 'push'
+        run: |
+          echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
+          bash hack/docker_build.sh ${{ steps.extract_branch.outputs.branch }}

.github/workflows/e2e-test.yml

@@ -0,0 +1,37 @@
+name: e2e
+
+on:
+  schedule:
+    # run e2e test every 4 hours
+    - cron: 0 */4 * * *
+  workflow_dispatch:
+jobs:
+  build:
+    name: Test
+    runs-on: ubuntu-latest
+    env:
+      GO111MODULE: on
+    steps:
+
+      - name: Set up Go 1.13
+        uses: actions/setup-go@v1
+        with:
+          go-version: 1.13
+        id: go
+
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+
+      - name: Downloading go dependencies
+        run: go mod vendor
+
+      - name: Create kind cluster
+        uses: helm/kind-action@v1.0.0-rc.1
+        with:
+          config: .github/workflows/kind/kind.yaml
+
+      - name: Deploy KubeSphere to Kind
+        run: KIND_CLUSTER_NAME=chart-testing hack/deploy-kubesphere.sh
+
+      - name: Run e2e testing
+        run: go test ./test/e2e

.github/workflows/kind/kind.yaml

@@ -0,0 +1,11 @@
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+nodes:
+- role: control-plane
+  image: kindest/node:v1.19.7
+  extraMounts:
+  - hostPath: /etc/localtime
+    containerPath: /etc/localtime
+  extraPortMappings:
+  - containerPort: 30881
+    hostPort: 9090

.github/workflows/nightly-builds.yml

@@ -0,0 +1,68 @@
+name: NightlyBuild
+
+on:
+  schedule:
+    # This is a UTC time
+    - cron: "0 16 * * *"
+  # Keep it only for test purpose, comment it once everything is ok
+  workflow_dispatch:
+
+jobs:
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    env:
+      GO111MODULE: on
+    steps:
+
+      - name: Set up Go 1.13
+        uses: actions/setup-go@v1
+        with:
+          go-version: 1.13
+        id: go
+
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Downloading go dependencies
+        run: go mod vendor
+
+      - name: Install kubebuilder
+        run: bash hack/install_kubebuilder.sh
+
+      - name: Build
+        run: make all
+
+      - name: Make OpenAPI Spec
+        run: make openapi
+
+      - name: Build and push docker images
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
+        run: |
+          echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
+          bash hack/docker_build.sh master
+
+          if [[ $? == 0 ]]; then
+            tag=nightly-$(date '+%Y%m%d')
+
+            docker tag kubespheredev/ks-apiserver kubespheredev/ks-apiserver:${tag}
+            docker tag kubespheredev/ks-controller-manager kubespheredev/ks-controller-manager:${tag}
+
+            docker push kubespheredev/ks-apiserver:${tag}
+            docker push kubespheredev/ks-controller-manager:${tag}
+          else
+            exit -1
+          fi
+
+      - name: slack
+        uses: 8398a7/action-slack@v3
+        env:
+          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+        with:
+          status: ${{ job.status }}
+          fields: repo,message,commit,author,action,eventName,ref,workflow,job,took
+        if: always()

.gitignore

@@ -19,14 +19,16 @@ bin/
 
 # Vscode files
 .vscode/
-
-tmp/
-
-apiserver.local.config
+__debug_bin
 
 # OSX trash
 .DS_Store
 api.json
 *.coverprofile
+cover.out
+coverage.txt
 
-kustomize/network/etcd
+kustomize/network/etcd
+apiserver.local.config
+tmp/
+kubesphere.yaml

.travis.yml

@@ -1,36 +0,0 @@
-services:
-  - docker
-
-language: go
-
-dist: xenial
-
-git:
-  depth: false
-
-go:
-  - "1.12.x"
-env:
-  - GO111MODULE=on
-cache:
-  directories:
-    - $HOME/gopath/pkg/mod
-
-before_script:
-  - docker --version
-  - bash hack/install_kubebuilder.sh
-
-script:
-  - diff -u <(echo -n) <(gofmt -d ./pkg ./cmd ./tools)
-  - make openapi
-  - make all
-
-install:
-  - go get golang.org/x/lint/golint
-
-deploy:
-  skip_cleanup: true
-  provider: script
-  script: bash hack/docker_build.sh 
-  on:
-    branch: master

CONTRIBUTING.md

@@ -1,221 +1,3 @@
-# Development Guide
+# Guide
 
-This document walks you through how to get started developing KubeSphere and development workflow.
-
-## Preparing the environment
-
-### Go
-
-KubeSphere development is based on [Kubernetes](https://github.com/kubernetes/kubernetes), both of them are written in [Go](http://golang.org/). If you don't have a Go development environment, please [set one up](http://golang.org/doc/code.html).
-
-| Kubernetes     | requires Go |
-|----------------|-------------|
-| 1.13+          | >= 1.12     |
-
-> Tips:
-> - Ensure your GOPATH and PATH have been configured in accordance with the Go
-environment instructions.
-> - It's recommended to install [macOS GNU tools](https://www.topbug.net/blog/2013/04/14/install-and-use-gnu-command-line-tools-in-mac-os-x) for Mac OS.
-
-### Docker
-
-KubeSphere components are often deployed as containers in Kubernetes. If you need to rebuild the KubeSphere components in the Kubernetes cluster, you will need to [install Docker](https://docs.docker.com/install/).
-
-
-### Dependency management
-
-KubeSphere uses [Go Modules](https://github.com/golang/go/wiki/Modules) to manage dependencies in the `vendor/` tree.
-
-#### Dependencies
-
-[kubesphere/kubesphere](https://github.com/kubesphere/kubesphere) repository contains the source code . If you're looking for its dependent components, they live in their own repositories since they can be individual and universal.
-
-- [Alert](https://github.com/kubesphere/alert): Alert is an enterprise-grade general-purpose high-performance alerting system.
-- [Notification](https://github.com/openpitrix/notification): Notification is an enterprise-grade general-purpose high-performance notification system, it provides email notification service for KubeSphere currently.
-- [OpenPitrix](https://github.com/openpitrix/openpitrix): Application management platform on multi-cloud environment, it provides application template and application management for KubeSphere currently.
-- [SonarQube](https://github.com/SonarSource/sonarqube): Integrated in KubeSphere DevOps, it provides the capability to not only show health of an application but also to highlight issues newly introduced.
-
-## Building KubeSphere on a local OS/shell environment
-
-### For Quick Taste Binary
-
-```bash
-mkdir ks-tmp
-cd ks-tmp
-echo 'module kubesphere' > go.mod
-echo 'replace (
-        github.com/Sirupsen/logrus v1.4.1 => github.com/sirupsen/logrus v1.4.1
-      	github.com/kiali/kiali => github.com/kubesphere/kiali v0.15.1-0.20190407071308-6b5b818211c3
-      	github.com/kubernetes-sigs/application => github.com/kubesphere/application v0.0.0-20190518133311-b9d9eb0b5cf7
-      )' >> go.mod
-
-GO111MODULE=on go get kubesphere.io/kubesphere@d649e3d0bbc64bfba18816c904819e4850d021e0
-GO111MODULE=on go build -o ks-apiserver kubesphere.io/kubesphere/cmd/ks-apiserver # build ks-apiserver
-GO111MODULE=on go build -o ks-apigateway kubesphere.io/kubesphere/cmd/ks-apigateway # build ks-apigateway
-GO111MODULE=on go build -o ks-controller-manager kubesphere.io/kubesphere/cmd/controller-manager # build ks-controller-manager
-GO111MODULE=on go build -o ks-iam kubesphere.io/kubesphere/cmd/ks-iam # build ks-iam
-```
-
-### For Building KubeSphere Images
-
-KubeSphere components are often deployed as a container in a kubernetes cluster, you may need to build a Docker image locally.
-
-1. Clone repo to local
-
-```bash
-git clone https://github.com/kubesphere/kubesphere.git
-```
-
-2. Run Docker command to build image
-
-```bash
-# $REPO is the docker registry to push to
-# $Tag is the tag name of the docker image
-# The full go build process will be executed in the Dockerfile, so you may need to set GOPROXY in it.
-docker build -f build/ks-apigateway/Dockerfile -t $REPO/ks-apigateway:$TAG .
-docker build -f build/ks-apiserver/Dockerfile -t $REPO/ks-apiserver:$TAG .
-docker build -f build/ks-iam/Dockerfile -t $REPO/ks-account:$TAG .
-docker build -f build/ks-controller-manager/Dockerfile -t $REPO/ks-controller-manager:$TAG .
-docker build -f ./pkg/db/Dockerfile -t $REPO/ks-devops:flyway-$TAG ./pkg/db/
-```
-
-### Test
-
-In the development process, it is recommended to use local Kubernetes clusters, such as [minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/), or to install an single-node [all-in-one](https://github.com/kubesphere/kubesphere#all-in-one) environment (Kubernetes-based) for quick testing.
-
-> Tip: It also supports to use Docker for Desktop ships with Kubernetes as the test environment.
-
-## Development Workflow
-
-![ks-workflow](docs/images/ks-workflow.png)
-
-### 1 Fork in the cloud
-
-1. Visit https://github.com/kubesphere/kubesphere
-2. Click `Fork` button to establish a cloud-based fork.
-
-### 2 Clone fork to local storage
-
-Per Go's [workspace instructions](https://golang.org/doc/code.html#Workspaces), place KubeSphere' code on your `GOPATH` using the following cloning procedure.
-
-1. Define a local working directory:
-
-```bash
-$ export working_dir=$GOPATH/src/kubesphere.io
-$ export user={your github profile name}
-```
-
-2. Create your clone locally:
-
-```bash
-$ mkdir -p $working_dir
-$ cd $working_dir
-$ git clone https://github.com/$user/kubesphere.git
-$ cd $working_dir/kubesphere
-$ git remote add upstream https://github.com/kubesphere/kubesphere.git
-
-# Never push to upstream master
-$ git remote set-url --push upstream no_push
-
-# Confirm that your remotes make sense:
-$ git remote -v
-```
-
-### 3 Keep your branch in sync
-
-```bash
-git fetch upstream
-git checkout master
-git rebase upstream/master
-```
-
-### 4 Add new features or fix issues
-
-Branch from it:
-
-```bash
-$ git checkout -b myfeature
-```
-
-Then edit code on the myfeature branch.
-
-**Test and build**
-
-Currently, make rules only contain simple checks such as vet, unit test, will add e2e tests soon.
-
-**Using KubeBuilder**
-
-- For Linux OS, you can download and execute this [KubeBuilder script](https://raw.githubusercontent.com/kubesphere/kubesphere/master/hack/install_kubebuilder.sh).
-
-- For MacOS, you can install KubeBuilder by following this [guide](https://book.kubebuilder.io/quick-start.html).
-
-**Run and test**
-
-```bash
-$ make all
-# Run every unit test
-$ make test
-```
-
-Run `make help` for additional information on these make targets.
-
-### 5 Development in new branch
-
-**Sync with upstream**
-
-After the test is completed, suggest you to keep your local in sync with upstream which can avoid conflicts.
-
-```
-# Rebase your the master branch of your local repo.
-$ git checkout master
-$ git rebase upstream/master
-
-# Then make your development branch in sync with master branch
-git checkout new_feature
-git rebase -i master
-```
-**Commit local changes**
-
-```bash
-$ git add <file>
-$ git commit -s -m "add your description"
-```
-
-### 6 Push to your folk
-
-When ready to review (or just to establish an offsite backup or your work), push your branch to your fork on github.com:
-
-```
-$ git push -f ${your_remote_name} myfeature
-```
-
-### 7 Create a PR
-
-- Visit your fork at https://github.com/$user/kubesphere
-- Click the` Compare & Pull Request` button next to your myfeature branch.
-- Check out the [pull request process](pull-request.md) for more details and advice.
-
-
-## CI/CD
-
-KubeSphere uses [Travis CI](https://travis-ci.org/) as a CI/CD tool.
-
-The components of KubeSphere need to be compiled and build include following:
-
-`ks-apiserver, ks-controller-manager, ks-account, ks-apigateway, ks-devops`
-
-After your PR is merged，Travis CI will compile the entire project and build the image, and push the image `kubespheredev/[component-name]:latest` to Dockerhub (e.g. `kubespheredev/ks-apiserver:latest`)
-
-## API Reference
-
-KubeSphere provides standard RESTFul API and detailed API documentations for developers, see [KubeSphere API Reference](https://docs.kubesphere.io/advanced-v2.0/zh-CN/api-reference/api-docs/) for more information.
-
-## Code conventions
-
-Please reference [Code conventions](https://github.com/kubernetes/community/blob/master/contributors/guide/coding-conventions.md) and follow with the rules.
-
-**Note:**
-
-> - All new packages and most new significant functionality must come with unit tests
-> - Comment your code in English, see [Go's commenting conventions
-](http://blog.golang.org/godoc-documenting-go-code)
+This [document](https://github.com/kubesphere/community) walks you through how to get started contributing KubeSphere.

Makefile

@@ -2,9 +2,6 @@
 # Use of this source code is governed by a Apache license
 # that can be found in the LICENSE file.
 
-# The binary to build 
-BIN ?= ks-apiserver
-
 # Produce CRDs that work back to Kubernetes 1.11 (no version conversion)
 CRD_OPTIONS ?= "crd:trivialVersions=true"
 
@@ -15,8 +12,6 @@ else
 GOBIN=$(shell go env GOBIN)
 endif
 
-
-IMG ?= kubespheredev/ks-apiserver
 OUTPUT_DIR=bin
 GOFLAGS=-mod=vendor
 define ALL_HELP_INFO
@@ -39,48 +34,41 @@ define ALL_HELP_INFO
 #           debugging tools like delve.
 endef
 .PHONY: all
-all: hypersphere ks-apiserver ks-apigateway ks-iam controller-manager
+all: test ks-apiserver ks-controller-manager
 
 # Build ks-apiserver binary
-ks-apiserver: test
+ks-apiserver: fmt vet
 	hack/gobuild.sh cmd/ks-apiserver
 
-# Build ks-apigateway binary
-ks-apigateway: test
-	hack/gobuild.sh cmd/ks-apigateway
-
-# Build ks-iam binary
-ks-iam: test
-	hack/gobuild.sh cmd/ks-iam
-
-# Build controller-manager binary
-controller-manager: test
+# Build ks-controller-manager binary
+ks-controller-manager: fmt vet
 	hack/gobuild.sh cmd/controller-manager
 
-# Build hypersphere binary
-hypersphere: test
-	hack/gobuild.sh cmd/hypersphere
+# Build e2e binary
+e2e: fmt vet
+	hack/build_e2e.sh test/e2e
 
 # Run go fmt against code 
-fmt: generate
+fmt:
 	gofmt -w ./pkg ./cmd ./tools ./api
 
+goimports:
+	@hack/update-goimports.sh
+
 # Run go vet against code
-vet: generate
+vet:
 	go vet ./pkg/... ./cmd/...
 
 # Generate manifests e.g. CRD, RBAC etc.
 manifests:
-	go run ./vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go all
+	go run ./vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go object:headerFile=./hack/boilerplate.go.txt paths=./pkg/apis/... rbac:roleName=controller-perms ${CRD_OPTIONS} output:crd:artifacts:config=config/crds
 
 deploy: manifests
 	kubectl apply -f config/crds
 	kustomize build config/default | kubectl apply -f -
 
-# generate will generate crds' deepcopy & go openapi structs
-# Futher more about go:genreate . https://blog.golang.org/generate
-generate:
-	go generate ./pkg/... ./cmd/...
+mockgen:
+	mockgen -package=openpitrix -source=pkg/simple/client/openpitrix/openpitrix.go -destination=pkg/simple/client/openpitrix/mock.go
 
 deepcopy:
 	GO111MODULE=on go install -mod=vendor k8s.io/code-generator/cmd/deepcopy-gen
@@ -89,16 +77,21 @@ deepcopy:
 openapi:
 	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./pkg/apis/tenant/v1alpha1 -p kubesphere.io/kubesphere/pkg/apis/tenant/v1alpha1 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list
 	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./pkg/apis/servicemesh/v1alpha2 -p kubesphere.io/kubesphere/pkg/apis/servicemesh/v1alpha2 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list
-	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/api/networking/v1,./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./pkg/apis/network/v1alpha1 -p kubesphere.io/kubesphere/pkg/apis/network/v1alpha1 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list
-	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./pkg/apis/devops/v1alpha1 -p kubesphere.io/kubesphere/pkg/apis/devops/v1alpha1 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list
+	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/api/networking/v1,./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./vendor/k8s.io/apimachinery/pkg/util/intstr,./pkg/apis/network/v1alpha1 -p kubesphere.io/kubesphere/pkg/apis/network/v1alpha1 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list
+	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./pkg/apis/devops/v1alpha1,./vendor/k8s.io/apimachinery/pkg/runtime,./vendor/k8s.io/api/core/v1 -p kubesphere.io/kubesphere/pkg/apis/devops/v1alpha1 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list
+	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./pkg/apis/cluster/v1alpha1,./vendor/k8s.io/apimachinery/pkg/runtime,./vendor/k8s.io/api/core/v1 -p kubesphere.io/kubesphere/pkg/apis/cluster/v1alpha1 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list
+	go run ./vendor/k8s.io/kube-openapi/cmd/openapi-gen/openapi-gen.go -O openapi_generated -i ./vendor/k8s.io/apimachinery/pkg/apis/meta/v1,./pkg/apis/devops/v1alpha3,./vendor/k8s.io/apimachinery/pkg/runtime -p kubesphere.io/kubesphere/pkg/apis/devops/v1alpha3 -h ./hack/boilerplate.go.txt --report-filename ./api/api-rules/violation_exceptions.list
 	go run ./tools/cmd/crd-doc-gen/main.go
+	go run ./tools/cmd/doc-gen/main.go
 # Build the docker image
 docker-build: all
-	docker build . -t ${IMG}
+	hack/docker_build.sh
+docker-build-no-test: ks-apiserver ks-controller-manager
+	hack/docker_build.sh
 
 # Run tests
 test: fmt vet
-	export KUBEBUILDER_CONTROLPLANE_START_TIMEOUT=1m; go test ./pkg/... ./cmd/... -coverprofile cover.out
+	export KUBEBUILDER_CONTROLPLANE_START_TIMEOUT=2m; go test ./pkg/... ./cmd/... -covermode=atomic -coverprofile=coverage.txt
 
 .PHONY: clean
 clean:

OWNERS

@@ -18,3 +18,6 @@ reviewers:
     - wnxn
     - zheng1
     - soulseen
+    - shaowenchen
+    - stoneshi-yunify
+    - linuxsuren

README.md

@@ -1,7 +1,9 @@
-# KubeSphere
+# KubeSphere Container Platform
+
 [![License](http://img.shields.io/badge/license-apache%20v2-blue.svg)](https://github.com/KubeSphere/KubeSphere/blob/master/LICENSE)
 [![Build Status](https://travis-ci.org/kubesphere/kubesphere.svg?branch=master)](https://travis-ci.org/kubesphere/kubesphere)
-[![KubeSphere release](https://img.shields.io/github/release/kubesphere/kubesphere.svg?color=release&label=release&logo=release&logoColor=release)](https://github.com/kubesphere/kubesphere/releases/tag/advanced-2.0.2)
+[![Go Report Card](https://goreportcard.com/badge/github.com/kubesphere/kubesphere)](https://goreportcard.com/report/github.com/kubesphere/kubesphere)
+[![KubeSphere release](https://img.shields.io/github/release/kubesphere/kubesphere.svg?color=release&label=release&logo=release&logoColor=release)](https://github.com/kubesphere/kubesphere/releases/tag/v3.0.0)
 
 ![logo](docs/images/kubesphere-logo.png)
 
@@ -11,151 +13,103 @@
 
 > English | [中文](README_zh.md)
 
-[KubeSphere](https://kubesphere.io/) is an enterprise-grade multi-tenant container management platform that built on [Kubernetes](https://kubernetes.io). It provides an easy-to-use UI for users to manage computing resources with a few clicks, which reduces the learning curve and empowers the DevOps teams. It greatly reduces the complexity of the daily work of development, testing, operation and maintenance, aiming to alleviate the pain points of Kubernetes' storage, network, security and ease of use, etc.
-
-
-## Screenshots
-
-> Note: See the [Screenshots](docs/screenshots.md) of KubeSphere to have a most intuitive understanding of KubeSphere dashboard and features.
+[KubeSphere](https://kubesphere.io/) is a **distributed operating system providing cloud native stack** with [Kubernetes](https://kubernetes.io) as its kernel, and aims to be plug-and-play architecture for third-party applications seamless integration to boost its ecosystem. KubeSphere is also a multi-tenant enterprise-grade container platform with full-stack automated IT operation and streamlined DevOps workflows. It provides developer-friendly wizard web UI, helping enterprises to build out a more robust and feature-rich platform, which includes most common functionalities needed for enterprise Kubernetes strategy, see [Feature List](#features) for details.
 
+The following screenshots give a close insight into KubeSphere. Please check [What is KubeSphere](https://kubesphere.io/docs/introduction/what-is-kubesphere/) for further information.
 
 <table>
   <tr>
-      <td width="50%" align="center"><b>KubeSphere Dashboard</b></td>
+      <td width="50%" align="center"><b>Workbench</b></td>
       <td width="50%" align="center"><b>Project Resources</b></td>
   </tr>
   <tr>
-     <td><img src="https://pek3b.qingstor.com/kubesphere-docs/png/20191112094014.png"/></td>
-     <td><img src="https://pek3b.qingstor.com/kubesphere-docs/png/20191112094426.png"/></td>
+     <td><img src="docs/images/console.png"/></td>
+     <td><img src="docs/images/project.png"/></td>
   </tr>
   <tr>
       <td width="50%" align="center"><b>CI/CD Pipeline</b></td>
-      <td width="50%" align="center"><b>Application Store</b></td>
+      <td width="50%" align="center"><b>App Store</b></td>
   </tr>
   <tr>
-     <td><img src="https://pek3b.qingstor.com/kubesphere-docs/png/20190925000712.png"/></td>
-     <td><img src="https://pek3b.qingstor.com/kubesphere-docs/png/20191112095006.png"/></td>
+     <td><img src="docs/images/cicd.png"/></td>
+     <td><img src="docs/images/app-store.png"/></td>
   </tr>
 </table>
 
-## Video on Youtube
+## Demo Environment
 
-[![KubeSphere](https://pek3b.qingstor.com/kubesphere-docs/png/20191112093503.png)](https://youtu.be/u5lQvhi_Xlc)
-
-## Features
-
-KubeSphere provides an easy-to-use console with the awesome user experience that allows you to quickly get started with a container management platform. KubeSphere provides and supports following core features:
-
-
-- Workload management
-- Service mesh (Istio-based)
-- DevOps
-- Source to Image
-- Multi-tenant management
-- Multi-dimensional and Multi-tenant Monitoring, Logging, Alerting, Notification
-- Service and network management
-- Application template and repository
-- Infrastructure management, image registry management
-- Integrate Harbor and GitLab
-- LB controller for Kubernetes on bare metal ([Porter](https://github.com/kubesphere/porter)), [cloud LB plugin](https://github.com/yunify/qingcloud-cloud-controller-manager)
-- Support GPU node
-
-
-It also supports multiple open source storage and high-performance cloud storage as the persistent storage services, as well as supports multiple open source network plugins.
-
-> Note: See this [document](https://docs.kubesphere.io/advanced-v2.0/zh-CN/introduction/features/) that elaborates on the KubeSphere features and services from a professional point of view.
-
-----
+Using the account `demo1 / Demo123` to log in the [demo environment](https://demo.kubesphere.io/). Please note the account is granted view access. You can also have a quick view of [KubeSphere Demo Video](https://youtu.be/u5lQvhi_Xlc).
 
 ## Architecture
 
-KubeSphere adopts the separation of front and back ends, each component is drawn in the architecture diagram below. KubeSphere can run anywhere from on-premise datacenter to any cloud to edge. In addition, it can be deployed on any Kubernetes distribution.
+KubeSphere uses a loosely-coupled architecture that separates the [frontend](https://github.com/kubesphere/console) from the [backend](https://github.com/kubesphere/kubesphere). External systems can access the components of the backend which are delivered as Docker containers through the REST APIs. See [Architecture](https://kubesphere.io/docs/introduction/architecture/) for details.
 
-![](https://pek3b.qingstor.com/kubesphere-docs/png/20190810073322.png)
+![Architecture](docs/images/architecture.png)
+
+## Features
+
+|Feature|Description|
+|---|---|
+| Provisioning Kubernetes Cluster|Support deploy Kubernetes on your infrastructure out of box, including online and air gapped installation|
+| Multi-cluster Management | Provide a centralized control plane to manage multiple Kubernetes Clusters, support application distribution across multiple clusters and cloud providers|
+| Kubernetes Resource Management | Provide web console for creating and managing Kubernetes resources, with powerful observability including monitoring, logging, events, alerting and notification |
+| DevOps System | Provide out-of-box CI/CD based on Jenkins, and offers automated workflow tools including binary-to-image (B2I) and source-to-image (S2I) |
+| Application Store | Provide application store for Helm-based applications, and offers application lifecycle management |
+| Service Mesh (Istio-based) | Provide fine-grained traffic management, observability and tracing for distributed microservice applications, provides visualization for traffic topology |
+| Rich Observability | Provide multi-dimensional monitoring metrics, and provides multi-tenant logging, events and [auditing](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/) management, support alerting and notification for both application and infrastructure |
+| Multi-tenant Management | Provide unified authentication with fine-grained roles and three-tier authorization system, supports AD/LDAP authentication |
+| Infrastructure Management | Support node management and monitoring, and supports adding new nodes for Kubernetes cluster |
+| Storage Support | Support GlusterFS, CephRBD, NFS, LocalPV (default), etc. open source storage solutions, provides CSI plugins to consume storage from cloud providers |
+| Network Support | Support Calico, Flannel, etc., provides [Network Policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) management, and load balancer plug-in [Porter](https://github.com/kubesphere/porter) for bare metal.|
+| GPU Support | Support add GPU node, support vGPU, enables running ML applications on Kubernetes, e.g. TensorFlow |
+
+Please see the [Feature and Benefits](https://kubesphere.io/docs/introduction/features/) for further information.
+
+----
 
 ## Latest Release
 
-KubeSphere 2.1.0 was released on **November 12nd, 2019**. See the [Release Notes For 2.1.0](https://kubesphere.io/docs/v2.1/zh-CN/release/release-v210/) to preview the updates.
+KubeSphere 3.0.0 is now generally available! See the [Release Notes For 3.0.0](https://kubesphere.io/docs/release/release-v300/) for the updates.
 
 ## Installation
 
-> Attention: Following section is only used for minimal installation by default, KubeSphere has decoupled some core components in v2.1.0, for more pluggable components installation, see `Enable Pluggable Components` below.
+KubeSphere can run anywhere from on-premise datacenter to any cloud to edge. In addition, it can be deployed on any version-compatible running Kubernetes cluster.
 
-### Deploy On Kubernetes
+### QuickStarts
 
-**Prerequisites**
+[Quickstarts](https://kubesphere.io/docs/quick-start/) include six hands-on lab exercises that help you quickly get started with KubeSphere.
 
-> - `Kubernetes version`： `1.13.0 ≤ K8s version < 1.16`;
-> - `Helm version` >= `2.10.0`，see [Install and Configure Helm in Kubernetes](https://devopscube.com/install-configure-helm-kubernetes/);
-> - CPU > 1 Core，Memory > 2 G;
-> - An existing Storage Class in your Kubernetes clusters, use `kubectl get sc` to verify it.
+### Installing on Existing Kubernetes Cluster
 
-When all Pods of KubeSphere are running, it means the installation is successsful. Then you can use `http://IP:30880` to access the dashboard with default account `admin/P@88w0rd`.
+- [Installing KubeSphere on Amazon EKS](https://v3-0.docs.kubesphere.io/docs/installing-on-kubernetes/hosted-kubernetes/install-kubesphere-on-eks/)
+- [Installing KubeSphere on Azure AKS](https://v3-0.docs.kubesphere.io/docs/installing-on-kubernetes/hosted-kubernetes/install-kubesphere-on-aks/)
+- [Installing KubeSphere on Google GKE](https://v3-0.docs.kubesphere.io/docs/installing-on-kubernetes/hosted-kubernetes/install-kubesphere-on-gke/)
+- [Installing KubeSphere on DigitalOcean Kubernetes](https://v3-0.docs.kubesphere.io/docs/installing-on-kubernetes/hosted-kubernetes/install-kubesphere-on-do/)
+- [Installing KubeSphere on Oracle OKE](https://v3-0.docs.kubesphere.io/docs/installing-on-kubernetes/hosted-kubernetes/install-kubesphere-on-oke/)
+- [Installing KubeSphere on Tencent TKE](https://v3-0.docs.kubesphere.io/docs/installing-on-kubernetes/hosted-kubernetes/install-ks-on-tencent-tke/)
+- [Installing KubeSphere on Huaweicloud CCE](https://v3-0.docs.kubesphere.io/docs/installing-on-kubernetes/hosted-kubernetes/install-ks-on-huawei-cce/)
 
-```yaml
-$ kubectl apply -f https://raw.githubusercontent.com/kubesphere/ks-installer/master/kubesphere-minimal.yaml
-```
+### Installing on Linux
 
+- [Installing KubeSphere on Azure VM](https://v3-0.docs.kubesphere.io/docs/installing-on-linux/public-cloud/install-ks-on-azure-vms/)
+- [Installing KubeSphere on VMware vSphere](https://v3-0.docs.kubesphere.io/docs/installing-on-linux/on-premises/install-kubesphere-on-vmware-vsphere/)
+- [Installing KubeSphere on QingCloud Instance](https://v3-0.docs.kubesphere.io/docs/installing-on-linux/public-cloud/kubesphere-on-qingcloud-instance/)
+- [Installing on Alibaba Cloud ECS](https://v3-0.docs.kubesphere.io/docs/installing-on-linux/public-cloud/install-kubesphere-on-ali-ecs/)
+- [Installing on Huaweicloud VM](https://v3-0.docs.kubesphere.io/docs/installing-on-linux/public-cloud/install-ks-on-huaweicloud-ecs/)
 
-### Deploy on Linux
+## Contributing, Support, Discussion, and Community
 
-- Operating Systems
-   - CentOS 7.5 (64 bit)
-   - Ubuntu 16.04/18.04 LTS (64 bit)
-   - Red Hat Enterprise Linux Server 7.4 (64 bit)
-   - Debian Stretch 9.5 (64 bit)
-- Hardware
-   - CPU：2 Core,  Memory：4 G, Disk Space：100 G
+We :heart: your contribution. The [community](https://github.com/kubesphere/community) walks you through how to get started contributing KubeSphere. The [development guide](https://github.com/kubesphere/community/tree/master/developer-guide/development) explains how to set up development environment.
 
-### All-in-One
+- [Slack Channel](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE)
+- [Youtube](https://www.youtube.com/channel/UCyTdUQUYjf7XLjxECx63Hpw)
+- [Follow us on Twitter](https://twitter.com/KubeSphere)
 
-For those who are new to KubeSphere and looking for the fastest way to install and experience the dashboard. Execute following commands to download and install KubeSphere in a single node.
+Please submit any KubeSphere bugs, issues, and feature requests to [KubeSphere GitHub Issue](https://github.com/kubesphere/kubesphere/issues).
 
-```bash
-$ curl -L https://kubesphere.io/download/stable/v2.1.0 > installer.tar.gz \
-&& tar -zxf installer.tar.gz && cd kubesphere-all-v2.1.0/scripts
-$ ./install.sh
-```
+## Who are using KubeSphere
 
-Choose `"1) All-in-one"` to trigger the installation. Generally, you can install it directly without any configuration..
-
-> Note: In a formal environment, it's highly recommended to install KubeSphere with Multi-Node Installation.
-
-### Enable Pluggable Components
-
-The above two methods is only used for minimal installation by default, execute following command to enable more pluggable components installation, make sure your cluster has enough CPU and memory in advance.
-
-```
-$ kubectl edit cm -n kubesphere-system ks-installer
-```
-
-## To start using KubeSphere
-
-### Quick Start
-
-KubeSphere provides 12 quick-start tutorials to walk you through the process and common manipulation, with a quick overview of the core features of KubeSphere that helps you to get familiar with it.
-
-- [Get Started - En](https://github.com/kubesphere/kubesphere.github.io/tree/master/blog/advanced-2.0/en)
-- [Get Started - 中](https://kubesphere.io/docs/advanced-v2.0/zh-CN/quick-start/quick-start-guide/)
-
-
-### Documentation
-
-- [KubeSphere Documentation (En/中) ](https://kubesphere.io/docs)
-- [API Documentation](https://kubesphere.io/docs/advanced-v2.0/zh-CN/api-reference/api-docs/)
-
-
-## To start developing KubeSphere
-
-The [development guide](CONTRIBUTING.md) hosts all information about building KubeSphere from source, git workflow, how to contribute code and how to test.
-
-## RoadMap
-
-Currently, KubeSphere has released the following 4 major editions. The future releases will include Multicluster, Big data, AI, SDN, etc.
-
-**Express Edition** => **v1.0.x** => **v2.0.x**  => **v2.1.0**
-
-![](https://pek3b.qingstor.com/kubesphere-docs/png/20190926000413.png)
+The [user case studies](https://kubesphere.io/case/) page includes the user list of the project. You can [submit a PR](https://github.com/kubesphere/kubesphere/blob/master/docs/powered-by-kubesphere.md) to add your institution name and homepage if you are using KubeSphere.
 
 ## Landscapes
 
@@ -167,16 +121,3 @@ KubeSphere is a member of CNCF and a <a href="https://www.cncf.io/certification/
 </a>, which enriches the <a href="https://landscape.cncf.io/landscape=observability-and-analysis&license=apache-license-2-0">CNCF CLOUD NATIVE Landscape.
 </a>
 </p>
-
-
-## Support, Discussion, and Community
-
-If you need any help with KubeSphere, please join us at [Slack Channel](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE).
-
-Please submit any KubeSphere bugs, issues, and feature requests to [KubeSphere GitHub Issue](https://github.com/kubesphere/kubesphere/issues).
-
-## Contributing to the project
-
-All members of the KubeSphere community must abide by [Code of Conduct](docs/code-of-conduct.md). Only by respecting each other can we develop a productive, collaborative community.
-
-How to submit a pull request to KubeSphere? See [Pull Request Instruction](docs/pull-requests.md).

README_zh.md

@@ -1,7 +1,8 @@
-# KubeSphere
+# KubeSphere 容器平台
+
 [![License](http://img.shields.io/badge/license-apache%20v2-blue.svg)](https://github.com/KubeSphere/KubeSphere/blob/master/LICENSE)
 [![Build Status](https://travis-ci.org/kubesphere/kubesphere.svg?branch=master)](https://travis-ci.org/kubesphere/kubesphere)
-[![KubeSphere release](https://img.shields.io/github/release/kubesphere/kubesphere.svg?color=release&label=release&logo=release&logoColor=release)](https://github.com/kubesphere/kubesphere/releases/tag/advanced-2.0.2)
+[![KubeSphere release](https://img.shields.io/github/release/kubesphere/kubesphere.svg?color=release&label=release&logo=release&logoColor=release)](https://github.com/kubesphere/kubesphere/releases/tag/v3.0.0)
 
 ![logo](docs/images/kubesphere-logo.png)
 
@@ -11,139 +12,104 @@
 
 > [English](README.md) | 中文
 
-[KubeSphere](https://kubesphere.io/) 是在 [Kubernetes](https://kubernetes.io) 之上构建的以**应用为中心的**多租户**容器管理平台**，支持部署和运行在**任何基础设施之上**，提供**简单易用的操作界面**以及**向导式操作**方式，在降低用户使用容器调度平台学习成本的同时，极大减轻开发、测试、运维的日常工作的复杂度，旨在解决 Kubernetes 本身存在的存储、网络、安全和易用性等痛点。帮助企业轻松应对**敏捷开发、自动化运维、应用快速交付、微服务治理、多租户管理、监控日志告警、服务与网络管理、镜像仓库**等业务场景。
+[KubeSphere](https://kubesphere.com.cn) 是在 [Kubernetes](https://kubernetes.io) 之上构建的面向云原生应用的 **容器混合云**，支持多云与多集群管理，提供全栈的 IT 自动化运维的能力，简化企业的 DevOps 工作流。KubeSphere 提供了运维友好的向导式操作界面，帮助企业快速构建一个强大和功能丰富的容器云平台。KubeSphere 愿景是打造一个基于 Kubernetes 的云原生分布式操作系统，它的架构可以很方便地与云原生生态进行即插即用（plug-and-play）的集成。
 
-KubeSphere 已大规模服务于社区用户，广泛地应用在以容器为中心的开发测试及生产环境，大量服务平稳地运行在 KubeSphere 之上。
-
-> 说明：KubeSphere 目前最新的版本为高级版 2.0.2，并且所有版本 100% 开源，关于 KubeSphere 更详细的介绍与说明请参阅 [产品介绍](https://docs.kubesphere.io/advanced-v2.0/zh-CN/introduction/intro/)。
-
-
-点击 [KubeSphere 快览](docs/screenshots.md) 快速查看 KubeSphere UI；
+KubeSphere 目前最新的版本为 3.0.0，所有版本 100% 开源，关于 KubeSphere 更详细的介绍与说明请参阅 [什么是 KubeSphere](https://kubesphere.com.cn/docs/introduction/what-is-kubesphere/)。
 
 <table>
   <tr>
-      <td width="50%" align="center"><b>KubeSphere Dashboard</b></td>
-      <td width="50%" align="center"><b>Project Resources</b></td>
+      <td width="50%" align="center"><b>工作台</b></td>
+      <td width="50%" align="center"><b>项目资源</b></td>
   </tr>
   <tr>
-     <td><img src="https://pek3b.qingstor.com/kubesphere-docs/png/20191112094014.png"/></td>
-     <td><img src="https://pek3b.qingstor.com/kubesphere-docs/png/20191112094426.png"/></td>
+     <td><img src="docs/images/console.png"/></td>
+     <td><img src="docs/images/project.png"/></td>
   </tr>
   <tr>
-      <td width="50%" align="center"><b>CI/CD Pipeline</b></td>
-      <td width="50%" align="center"><b>Application Store</b></td>
+      <td width="50%" align="center"><b>CI/CD 流水线</b></td>
+      <td width="50%" align="center"><b>应用商店</b></td>
   </tr>
   <tr>
-     <td><img src="https://pek3b.qingstor.com/kubesphere-docs/png/20190925000712.png"/></td>
-     <td><img src="https://pek3b.qingstor.com/kubesphere-docs/png/20191112095006.png"/></td>
+     <td><img src="docs/images/cicd.png"/></td>
+     <td><img src="docs/images/app-store.png"/></td>
   </tr>
 </table>
 
-## Video on Youtube
+## 快速体验
 
-[![KubeSphere](https://pek3b.qingstor.com/kubesphere-docs/png/20191112093503.png)](https://youtu.be/u5lQvhi_Xlc)
-
-## 核心功能
-
-- Kubernetes 资源管理：纳管多种类型的 K8s 资源，提供易用友好的向导式 UI
-- 应用编排与管理：包括**一键部署应用**、**Helm Chart 可视化管理**、**应用生命周期管理**，后续将支持计量计费
-- 微服务治理：基于 Istio 提供可视化无代码侵入的**灰度发布、熔断、流量管控、Tracing**，兼容**Spring Cloud & Dubbo**
-- 一站式 DevOps：提供**可视化编辑 CI/CD 流水线**，包括从开发测试到持续部署上线的**全流程管理**，提供 [S2i](https://kubesphere.io/docs/v2.0/zh-CN/quick-start/source-to-image/)、[B2i](https://kubesphere.io/docs/v2.1/zh-CN/quick-start/b2i-war/)
-- 多租户管理：提供基于角色的细粒度 [多租户统一认证](https://kubesphere.io/docs/v2.1/zh-CN/multi-tenant/intro/)，支持**对接企业 LDAP/AD**，提供多层级的权限管理满足多组织架构的企业用户
-- 日志查询与收集：提供基于多租户和多维度的 [日志查询](https://kubesphere.io/docs/v2.1/zh-CN/toolbox/log-search/)，并支持快速对接多种日志收集平台
-- 多维度监控：提供集群与应用级别多项监控指标，提供按节点、企业空间、项目等资源用量的排行
-- 多租户告警系统：支持基于多租户、多维度的告警，提供丰富的监控告警指标，可自定义告警策略，支持邮件通知
-- 基础设施管理：提供主机管理、存储类型管理、CPU 与内存等资源配额管理
-- 支持多种网络方案：支持 Calico、Flannel，提供面向物理部署 Kubernetes 的 LB 插件 [Porter](https://github.com/kubesphere/porter) 和云上[负载均衡器插件](https://github.com/yunify/qingcloud-cloud-controller-manager)
-- 支持多种存储：支持 GlusterFS、CephRBD、NFS，支持 [企业级分布式存储 NeonSAN](https://www.qingcloud.com/products/qingstor-neonsan/) 和 [QingCloud 云平台块存储](https://docs.qingcloud.com/product/storage/volume/)
-- 支持 GPU 节点
-
-
-> 更多详细的功能解读与说明，请查阅 [产品功能](https://kubesphere.io/docs/v2.1/zh-CN/introduction/features/)。
-
-----
+使用体验账号 `demo1 / Demo123` 登录 [Demo 环境](https://demo.kubesphere.io/)，该账号仅授予了 view 权限，建议自行安装体验完整的管理功能。您还可以访问 Youtube 查看 [KubeSphere Demo 视频](https://youtu.be/u5lQvhi_Xlc)。
 
 ## 架构
 
-KubeSphere 采用了前后端分离的架构设计，后端的各个功能组件可通过 REST API 对接外部系统，KubeSphere 可以运行在任何 Kubernetes、私有云、公有云、VM 或物理环境之上。
+KubeSphere 采用了前后端分离的架构设计，后端的各个功能组件可通过 REST API 对接外部系统，详见 [架构说明](https://kubesphere.com.cn/docs/zh-CN/introduction/architecture/)。本仓库仅包含后端代码，前端代码参考 [Console 项目](https://github.com/kubesphere/console)。
 
-![](docs/images/architecture-zh.png)
+![Architecture](docs/images/architecture.png)
+
+## 核心功能
+
+|功能 |介绍 |
+| --- | ---|
+|多云与多集群管理|提供多云与多集群的中央管理面板，支持集群导入，支持应用在多云与多集群一键分发|
+| Kubernetes 集群搭建与运维 | 支持在线 & 离线安装、升级与扩容 K8s 集群，支持安装 “云原生全家桶” |
+| Kubernetes 资源可视化管理 | 可视化纳管原生 Kubernetes 资源，支持向导式创建与管理 K8s 资源 |
+| 基于 Jenkins 的 DevOps 系统 | 支持图形化与脚本两种方式构建 CI/CD 流水线，内置 Source to Image（S2I）和 Binary to Image（B2I）等 CD 工具 |
+| 应用商店与应用生命周期管理 | 提供应用商店，内置 Redis、MySQL 等 15 个常用应用，支持应用的生命周期管理 |
+| 基于 Istio 的微服务治理 (Service Mesh) | 提供可视化无代码侵入的 **灰度发布、熔断、流量治理与流量拓扑、分布式 Tracing** |
+| 多租户管理 | 提供基于角色的细粒度多租户统一认证，支持 **对接企业 LDAP/AD**，提供多层级的权限管理 |
+| 丰富的可观察性功能 | 提供集群/工作负载/Pod/容器等多维度的监控，提供基于多租户的日志查询与日志收集，支持节点与应用层级的告警与通知 |
+|基础设施管理|支持 Kubernetes 节点管理，支持节点扩容与集群升级，提供基于节点的多项监控指标与告警规则 |
+| 存储管理 | 支持对接 Ceph、GlusterFS、NFS、Local PV，支持可视化运维管理 PVC、StorageClass，提供 CSI 插件对接云平台存储 |
+| 网络管理 | 提供租户网络隔离与 K8s [Network Policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) 管理，支持 Calico、Flannel，提供 [Porter LB](https://github.com/kubesphere/porter) 用于暴露物理环境 K8s 集群的 LoadBalancer 服务 |
+| GPU support | 集群支持添加 GPU 与 vGPU，可运行 TensorFlow 等 ML 框架 |
+
+以上功能说明详见 [产品功能](https://kubesphere.com.cn/docs/zh-CN/introduction/features/)。
+
+----
 
 ## 最新发布
 
-KubeSphere 2.1.0 已于 2019 年 11 月 12 日 正式发布，点击 [Release Notes For 2.1.0](https://kubesphere.io/docs/v2.1/zh-CN/release/release-v210/) 查看 2.1.0 版本的更新详情。
+KubeSphere 3.0.0 已于 2020 年 8 月 31 日正式 GA！点击 [Release Notes For 3.0.0](https://kubesphere.com.cn/docs/release/release-v300/) 查看 3.0.0 版本的更新详情。
 
-## 快速安装
-
-### 部署在 Linux
-
-- 操作系统
-   - CentOS 7.5 (64 bit)
-   - Ubuntu 16.04/18.04 LTS (64 bit)
-   - Red Hat Enterprise Linux Server 7.4 (64 bit)
-   - Debian Stretch 9.5 (64 bit)
-- 配置规格（最低）
-   - CPU：2 Core， 内存：4 G， 硬盘：100 G
-
-#### All-in-One
-
-[All-in-One](https://kubesphere.io/docs/v2.1/zh-CN/installation/all-in-one/): 对于首次接触 KubeSphere 高级版的用户，想寻找一个最快安装和体验 KubeSphere 高级版核心功能的方式，All-in-one 模式支持一键安装 KubeSphere 至一台目标机器，建议使用干净的机器安装。
-
-```bash
-$ curl -L https://kubesphere.io/download/stable/v2.1.0 > installer.tar.gz \
-&& tar -zxf installer.tar.gz && cd kubesphere-all-v2.1.0/scripts
-$ ./install.sh
-```
-
-直接选择 `"1) All-in-one"` 即可开始快速安装。
-
-> 注意：All-in-One 仅适用于**测试体验**，**正式环境** 安装和使用请参考 [安装说明](https://kubesphere.io/docs/v2.1/zh-CN/installation/intro/#%E6%AD%A3%E5%BC%8F%E7%8E%AF%E5%A2%83%E5%AE%89%E8%A3%85)。
-
-### 部署在 Kubernetes
-
-**前提条件**
-
-> - `Kubernetes` 版本： `1.13.0 ≤ K8s version < 1.16`；
-> - `Helm`，版本 `>= 2.10.0`，且已安装了 Tiller，参考 [如何安装与配置 Helm](https://devopscube.com/install-configure-helm-kubernetes/)；
-> - 集群的可用 CPU > 1 C，可用内存 > 2 G；且集群能够访问外网
-> - 集群已有存储类型（StorageClass）；
-
-可参考 [前提条件](https://kubesphere.io/docs/v2.1/zh-CN/installation/prerequisites/) 验证，若待安装的环境满足以上条件则可以开始部署 KubeSphere，当 KubeSphere 的所有 Pod 都为 RRunning 则说明安装成功。使用 `http://IP:30880` 访问 Dashboard，默认账号为 `admin/P@88w0rd`。
-
-```yaml
-$ kubectl apply -f https://raw.githubusercontent.com/kubesphere/ks-installer/master/kubesphere-minimal.yaml
-```
-
-注意，以上两种安装方式默认是 **最小化安装**，由于 2.1.0 已对大部分功能组件解耦，实现了功能组件的可插拔，因此可在 **安装完成后** 执行以下命令开启可插拔功能组件的安装，开启安装前确认您的机器资源已符合 [资源最低要求](https://kubesphere.io/docs/v2.1/zh-CN/installation/intro/#%E5%8F%AF%E6%8F%92%E6%8B%94%E5%8A%9F%E8%83%BD%E7%BB%84%E4%BB%B6%E5%88%97%E8%A1%A8)。
-
-```
-$ kubectl edit cm -n kubesphere-system ks-installer
-```
-
-## 开始使用 KubeSphere
+## 安装 3.0.0
 
 ### 快速入门
 
-[KubeSphere 快速入门](https://kubesphere.io/docs/v2.1/zh-CN/quick-start/quick-start-guide/) 通过 14 个 Step-by-Step 的快速入门的示例教程帮助您了解 KubeSphere 容器平台的基本使用流程，带您快速上手 KubeSphere。
+[快速入门系列](https://kubesphere.com.cn/docs/quick-start/) 提供了快速安装与入门示例，供初次安装体验参考。
 
-### 文档
+### 在已有 Kubernetes 之上安装 KubeSphere
 
-- [KubeSphere 文档中心 (En/中) ](https://kubesphere.io/docs/)
-- [API 文档](https://kubesphere.io/docs/advanced-v2.0/zh-CN/api-reference/api-docs/)
+- [基于 Kubernetes 的安装介绍](https://kubesphere.com.cn/docs/installing-on-kubernetes/introduction/overview/)
+- [在阿里云 ACK 安装 KubeSphere](https://kubesphere.com.cn/forum/d/1745-kubesphere-v3-0-0-dev-on-ack)
+- [在腾讯云 TKE 安装 KubeSphere](https://kubesphere.com.cn/docs/installing-on-kubernetes/hosted-kubernetes/install-ks-on-tencent-tke/)
+- [在华为云 CCE 安装 KubeSphere](https://kubesphere.com.cn/docs/installing-on-kubernetes/hosted-kubernetes/install-ks-on-huawei-cce/)
+- [在 AWS EKS 安装 KubeSphere](https://kubesphere.com.cn/docs/installing-on-kubernetes/hosted-kubernetes/install-kubesphere-on-eks/)
+- [在 Google GKE 安装 KubeSphere](https://kubesphere.com.cn/docs/installing-on-kubernetes/hosted-kubernetes/install-kubesphere-on-gke/)
+- [在 Azure AKS 安装 KubeSphere](https://kubesphere.com.cn/docs/installing-on-kubernetes/hosted-kubernetes/install-kubesphere-on-aks/)
+- [在 DigitalOcean 安装 KubeSphere](https://kubesphere.com.cn/docs/installing-on-kubernetes/hosted-kubernetes/install-kubesphere-on-do/)
+- [在 Oracle OKE 安装 KubeSphere](https://kubesphere.com.cn/docs/installing-on-kubernetes/hosted-kubernetes/install-kubesphere-on-oke/)
 
+### 基于 Linux 安装 KubeSphere
 
-## 开发 KubeSphere
+- [多节点安装介绍（以三节点为例）](https://kubesphere.com.cn/docs/installing-on-linux/introduction/multioverview/)
+- [在 VMware vSphere 安装高可用集群](https://kubesphere.com.cn/docs/installing-on-linux/on-premises/install-kubesphere-on-vmware-vsphere/)
+- [在青云QingCloud 安装高可用集群](https://kubesphere.com.cn/docs/installing-on-linux/public-cloud/kubesphere-on-qingcloud-instance/)
+- [在阿里云 ECS 部署高可用集群](https://kubesphere.com.cn/docs/installing-on-linux/public-cloud/install-kubesphere-on-ali-ecs/)
 
-[开发指南](CONTRIBUTING.md) 详细说明了如何从源码编译、KubeSphere 的 GitHub 工作流、如何贡献代码以及如何测试等。
+- [在华为云 VM 部署高可用集群](https://kubesphere.com.cn/docs/installing-on-linux/public-cloud/install-kubesphere-on-huaweicloud-ecs/)
+- [在 Azure VM 安装高可用集群](https://kubesphere.com.cn/docs/installing-on-linux/public-cloud/install-kubesphere-on-azure-vms/)
 
+## 技术社区
 
-## 路线图
+[KubeSphere 社区](https://github.com/kubesphere/community) 包含所有社区的信息，包括如何开发，兴趣小组(SIG)等。比如[开发指南](https://github.com/kubesphere/community/tree/master/developer-guide/development) 详细说明了如何从源码编译、KubeSphere 的 GitHub 工作流、如何贡献代码以及如何测试等。
 
-目前，KubeSphere 已发布了 4 个大版本和 3 个小版本，所有版本都是完全开源的，为 KubeSphere 社区用户提供服务。
+- [中文论坛](https://kubesphere.com.cn/forum/)
+- [Slack Channel](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE)
+- [社区微信群（见官网底部）](https://kubesphere.com.cn/)
+- [Bug 与建议反馈（GitHub Issue）](https://github.com/kubesphere/kubesphere/issues)
 
-**Express Edition** => **v1.0.x** => **v2.0.x**  => **v2.1.0**
+## 谁在使用 KubeSphere
 
-![](https://pek3b.qingstor.com/kubesphere-docs/png/20190926000514.png)
+[Powered by KubeSphere](https://kubesphere.com.cn/case/) 列出了哪些企业在使用 KubeSphere，如果您所在的企业已安装使用了 KubeSphere，欢迎[提交 PR](https://github.com/kubesphere/kubesphere/blob/master/docs/powered-by-kubesphere.md)。
 
 ## Landscapes
 
@@ -155,18 +121,3 @@ KubeSphere 是 CNCF 基金会成员并且通过了 <a href="https://www.cncf.io/
 </a>，进一步丰富了 <a href="https://landscape.cncf.io/landscape=observability-and-analysis&license=apache-license-2-0">CNCF 云原生的生态。
 </a>
 </p>
-
-## 技术社区
-
-- [Slack Channel](https://join.slack.com/t/kubesphere/shared_invite/enQtNTE3MDIxNzUxNzQ0LTZkNTdkYWNiYTVkMTM5ZThhODY1MjAyZmVlYWEwZmQ3ODQ1NmM1MGVkNWEzZTRhNzk0MzM5MmY4NDc3ZWVhMjE)
-
-- [技术论坛](https://kubesphere.io/forum/)
-
-- 微信群 (与工程师和用户们交流云原生技术，请备注 “公司 - 名字”）
-
-<img width="150px" src="https://pek3b.qingstor.com/kubesphere-docs/png/20190902002055.png" />
-
-
-## Bug 与建议反馈
-
-KubeSphere 的日益完善与快速发展离不开社区用户的支持，KubeSphere 也一直在反哺社区，为开源用户提供更多的支持。若您安装使用时有任何建议问题、反馈或发现的 Bug，欢迎在 [GitHub Issue](https://github.com/kubesphere/kubesphere/issues) 提交 Issue。

api/api-rules/violation_exceptions.list

@@ -1,3 +1,91 @@
+API rule violation: list_type_missing,./pkg/apis/devops/v1alpha3,DevOpsProjectList,Items
+API rule violation: list_type_missing,./pkg/apis/devops/v1alpha3,NoScmPipeline,Parameters
+API rule violation: list_type_missing,./pkg/apis/devops/v1alpha3,PipelineList,Items
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIGroup,ServerAddressByClientCIDRs
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIGroup,Versions
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIGroupList,Groups
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIResource,Categories
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIResource,ShortNames
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIResourceList,APIResources
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIVersions,ServerAddressByClientCIDRs
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,APIVersions,Versions
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,CreateOptions,DryRun
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,DeleteOptions,DryRun
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,FieldsV1,Raw
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,LabelSelector,MatchExpressions
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,LabelSelectorRequirement,Values
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,List,Items
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,ObjectMeta,Finalizers
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,ObjectMeta,ManagedFields
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,ObjectMeta,OwnerReferences
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,PartialObjectMetadataList,Items
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,PatchOptions,DryRun
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,RootPaths,Paths
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,StatusDetails,Causes
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,Table,ColumnDefinitions
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,Table,Rows
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,TableRow,Cells
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,TableRow,Conditions
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/apis/meta/v1,UpdateOptions,DryRun
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/runtime,RawExtension,Raw
+API rule violation: list_type_missing,k8s.io/apimachinery/pkg/runtime,Unknown,Raw
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,BitbucketServerSource,ApiUri
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,BitbucketServerSource,CloneOption
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,BitbucketServerSource,CredentialId
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,BitbucketServerSource,DiscoverBranches
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,BitbucketServerSource,DiscoverPRFromForks
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,BitbucketServerSource,DiscoverPRFromOrigin
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,BitbucketServerSource,DiscoverTags
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,BitbucketServerSource,RegexFilter
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,BitbucketServerSource,ScmId
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,DiscarderProperty,DaysToKeep
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,DiscarderProperty,NumToKeep
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitSource,CloneOption
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitSource,CredentialId
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitSource,DiscoverBranches
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitSource,DiscoverTags
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitSource,RegexFilter
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitSource,ScmId
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GithubSource,ApiUri
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GithubSource,CloneOption
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GithubSource,CredentialId
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GithubSource,DiscoverBranches
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GithubSource,DiscoverPRFromForks
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GithubSource,DiscoverPRFromOrigin
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GithubSource,DiscoverTags
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GithubSource,RegexFilter
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GithubSource,ScmId
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitlabSource,ApiUri
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitlabSource,CloneOption
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitlabSource,CredentialId
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitlabSource,DiscoverBranches
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitlabSource,DiscoverPRFromForks
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitlabSource,DiscoverPRFromOrigin
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitlabSource,DiscoverTags
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitlabSource,RegexFilter
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitlabSource,ScmId
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,GitlabSource,ServerName
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchJobTrigger,CreateActionJobsToTrigger
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchJobTrigger,DeleteActionJobsToTrigger
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchPipeline,BitbucketServerSource
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchPipeline,GitHubSource
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchPipeline,GitSource
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchPipeline,GitlabSource
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchPipeline,MultiBranchJobTrigger
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchPipeline,ScriptPath
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchPipeline,SingleSvnSource
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchPipeline,SourceType
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchPipeline,SvnSource
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,MultiBranchPipeline,TimerTrigger
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,NoScmPipeline,DisableConcurrent
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,NoScmPipeline,RemoteTrigger
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,NoScmPipeline,TimerTrigger
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,Parameter,DefaultValue
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,PipelineSpec,MultiBranchPipeline
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,SingleSvnSource,CredentialId
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,SingleSvnSource,ScmId
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,SvnSource,CredentialId
+API rule violation: names_match,./pkg/apis/devops/v1alpha3,SvnSource,ScmId
 API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,APIResourceList,APIResources
 API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,Duration,Duration
 API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,InternalEvent,Object
@@ -5,4 +93,6 @@ API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,InternalEve
 API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,MicroTime,Time
 API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,StatusCause,Type
 API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,Time,Time
-API rule violation: names_match,kubesphere.io/kubesphere/pkg/apis/devops/v1alpha1,S2iBinarySpec,MD5
+API rule violation: names_match,k8s.io/apimachinery/pkg/runtime,Unknown,ContentEncoding
+API rule violation: names_match,k8s.io/apimachinery/pkg/runtime,Unknown,ContentType
+API rule violation: names_match,k8s.io/apimachinery/pkg/runtime,Unknown,Raw

build/hypersphere/Dockerfile

@@ -1,18 +0,0 @@
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
-# that can be found in the LICENSE file.
-
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
-# that can be found in the LICENSE file.
-FROM golang:1.12 as hypersphere-builder
-
-COPY / /go/src/kubesphere.io/kubesphere
-
-WORKDIR /go/src/kubesphere.io/kubesphere
-RUN CGO_ENABLED=0 GO111MODULE=on GOOS=linux GOARCH=amd64 GOFLAGS=-mod=vendor go build -i -ldflags '-w -s' -o hypersphere cmd/hypersphere/hypersphere.go
-
-FROM alpine:3.9
-RUN apk add --update ca-certificates && update-ca-certificates
-COPY --from=hypersphere-builder /go/src/kubesphere.io/kubesphere/hypersphere /usr/local/bin/
-CMD ["sh"]

build/ks-apigateway/Dockerfile

@@ -1,20 +0,0 @@
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
-# that can be found in the LICENSE file.
-
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
-# that can be found in the LICENSE file.
-
-FROM golang:1.12 as ks-apigateway-builder
-
-COPY / /go/src/kubesphere.io/kubesphere
-WORKDIR /go/src/kubesphere.io/kubesphere
-RUN CGO_ENABLED=0 GO111MODULE=on GOOS=linux GOARCH=amd64 GOFLAGS=-mod=vendor go build -i -ldflags '-w -s' -o ks-apigateway cmd/ks-apigateway/apiserver.go && \
-    go run tools/cmd/doc-gen/main.go --output=install/swagger-ui/api.json
-
-FROM alpine:3.9
-RUN apk add --update ca-certificates && update-ca-certificates
-COPY --from=ks-apigateway-builder /go/src/kubesphere.io/kubesphere/ks-apigateway /usr/local/bin/
-COPY --from=ks-apigateway-builder /go/src/kubesphere.io/kubesphere/install/swagger-ui /var/static/swagger-ui
-CMD ["sh"]

build/ks-apiserver/Dockerfile

@@ -1,18 +1,23 @@
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
+# Copyright 2020 The KubeSphere Authors. All rights reserved.
+# Use of this source code is governed by an Apache license
 # that can be found in the LICENSE file.
+FROM alpine:3.11
 
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
-# that can be found in the LICENSE file.
-FROM golang:1.12 as ks-apiserver-builder
+ARG HELM_VERSION=v3.5.2
 
-COPY / /go/src/kubesphere.io/kubesphere
+RUN apk add --no-cache ca-certificates
+# install helm
+RUN wget https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz && \
+    tar xvf helm-${HELM_VERSION}-linux-amd64.tar.gz && \
+    rm helm-${HELM_VERSION}-linux-amd64.tar.gz && \
+    mv linux-amd64/helm /usr/bin/ && \
+    rm -rf linux-amd64
+# To speed up building process, we copy binary directly from make
+# result instead of building it again, so make sure you run the 
+# following command first before building docker image
+#   make ks-apiserver
+#
+COPY  /bin/cmd/ks-apiserver /usr/local/bin/
 
-WORKDIR /go/src/kubesphere.io/kubesphere
-RUN CGO_ENABLED=0 GO111MODULE=on GOOS=linux GOARCH=amd64 GOFLAGS=-mod=vendor go build -i -ldflags '-w -s' -o ks-apiserver cmd/ks-apiserver/apiserver.go
-
-FROM alpine:3.9
-RUN apk add --update ca-certificates && update-ca-certificates
-COPY --from=ks-apiserver-builder /go/src/kubesphere.io/kubesphere/ks-apiserver /usr/local/bin/
+EXPOSE 9090
 CMD ["sh"]

build/ks-controller-manager/Dockerfile

@@ -1,18 +1,26 @@
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
+# Copyright 2020 The KubeSphere Authors. All rights reserved.
+# Use of this source code is governed by an Apache license
 # that can be found in the LICENSE file.
+FROM alpine:3.11
 
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
-# that can be found in the LICENSE file.
-FROM golang:1.12 as controller-manager-builder
+ARG HELM_VERSION=v3.5.2
+ARG KUSTOMIZE_VERSION=v4.1.0
 
-COPY / /go/src/kubesphere.io/kubesphere
-WORKDIR /go/src/kubesphere.io/kubesphere
+RUN apk add --no-cache ca-certificates
+# install helm
+RUN wget https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz && \
+    tar xvf helm-${HELM_VERSION}-linux-amd64.tar.gz && \
+    rm helm-${HELM_VERSION}-linux-amd64.tar.gz && \
+    mv linux-amd64/helm /usr/bin/ && \
+    rm -rf linux-amd64
+# install kustomize
+RUN wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${KUSTOMIZE_VERSION}/kustomize_${KUSTOMIZE_VERSION}_linux_amd64.tar.gz && \
+    tar xvf kustomize_${KUSTOMIZE_VERSION}_linux_amd64.tar.gz && \
+    rm kustomize_${KUSTOMIZE_VERSION}_linux_amd64.tar.gz && \
+    mv kustomize /usr/bin
+    
+COPY  /bin/cmd/controller-manager /usr/local/bin/
 
-RUN CGO_ENABLED=0 GO111MODULE=on GOOS=linux GOARCH=amd64 GOFLAGS=-mod=vendor go build --ldflags "-extldflags -static" -o controller-manager ./cmd/controller-manager/
+EXPOSE 8443 8080
 
-FROM alpine:3.7
-RUN apk add --update ca-certificates && update-ca-certificates
-COPY --from=controller-manager-builder /go/src/kubesphere.io/kubesphere/controller-manager /usr/local/bin/
-CMD controller-manager
+CMD ["sh"]

build/ks-iam/Dockerfile

@@ -1,18 +0,0 @@
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
-# that can be found in the LICENSE file.
-
-# Copyright 2018 The KubeSphere Authors. All rights reserved.
-# Use of this source code is governed by a Apache license
-# that can be found in the LICENSE file.
-FROM golang:1.12 as ks-iam-builder
-
-COPY / /go/src/kubesphere.io/kubesphere
-
-WORKDIR /go/src/kubesphere.io/kubesphere
-RUN CGO_ENABLED=0 GO111MODULE=on GOOS=linux GOARCH=amd64 GOFLAGS=-mod=vendor go build -i -ldflags '-w -s' -o ks-iam cmd/ks-iam/apiserver.go
-
-FROM alpine:3.9
-RUN apk add --update ca-certificates && update-ca-certificates
-COPY --from=ks-iam-builder /go/src/kubesphere.io/kubesphere/ks-iam /usr/local/bin/
-CMD ["sh"]

build/ks-network/Dockerfile

@@ -1,4 +0,0 @@
-FROM gcr.io/distroless/static:latest
-WORKDIR /
-COPY ks-network .
-ENTRYPOINT ["/ks-network"]

cmd/controller-manager/app/controllers.go

@@ -1,154 +1,306 @@
 /*
+Copyright 2019 The KubeSphere Authors.
 
- Copyright 2019 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
 
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
-     http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
+    http://www.apache.org/licenses/LICENSE-2.0
 
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
 */
+
 package app
 
 import (
-	"k8s.io/client-go/informers"
-	"k8s.io/client-go/kubernetes"
-	"k8s.io/client-go/rest"
-	"kubesphere.io/kubesphere/pkg/controller/application"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/client-go/tools/cache"
+	"k8s.io/klog"
+	"sigs.k8s.io/controller-runtime/pkg/manager"
+	"sigs.k8s.io/kubefed/pkg/controller/util"
+
+	iamv1alpha2 "kubesphere.io/kubesphere/pkg/apis/iam/v1alpha2"
+	authoptions "kubesphere.io/kubesphere/pkg/apiserver/authentication/options"
+	"kubesphere.io/kubesphere/pkg/controller/certificatesigningrequest"
+	"kubesphere.io/kubesphere/pkg/controller/cluster"
+	"kubesphere.io/kubesphere/pkg/controller/clusterrolebinding"
 	"kubesphere.io/kubesphere/pkg/controller/destinationrule"
+	"kubesphere.io/kubesphere/pkg/controller/devopscredential"
+	"kubesphere.io/kubesphere/pkg/controller/devopsproject"
+	"kubesphere.io/kubesphere/pkg/controller/globalrole"
+	"kubesphere.io/kubesphere/pkg/controller/globalrolebinding"
+	"kubesphere.io/kubesphere/pkg/controller/group"
+	"kubesphere.io/kubesphere/pkg/controller/groupbinding"
 	"kubesphere.io/kubesphere/pkg/controller/job"
+	"kubesphere.io/kubesphere/pkg/controller/loginrecord"
+	"kubesphere.io/kubesphere/pkg/controller/network/ippool"
+	"kubesphere.io/kubesphere/pkg/controller/network/nsnetworkpolicy"
+	"kubesphere.io/kubesphere/pkg/controller/network/nsnetworkpolicy/provider"
+	"kubesphere.io/kubesphere/pkg/controller/notification"
+	"kubesphere.io/kubesphere/pkg/controller/pipeline"
 	"kubesphere.io/kubesphere/pkg/controller/s2ibinary"
 	"kubesphere.io/kubesphere/pkg/controller/s2irun"
+	"kubesphere.io/kubesphere/pkg/controller/storage/capability"
 	"kubesphere.io/kubesphere/pkg/controller/storage/expansion"
-
-	//"kubesphere.io/kubesphere/pkg/controller/job"
+	"kubesphere.io/kubesphere/pkg/controller/user"
 	"kubesphere.io/kubesphere/pkg/controller/virtualservice"
-	"sigs.k8s.io/controller-runtime/pkg/manager"
-	"time"
-
-	logf "sigs.k8s.io/controller-runtime/pkg/runtime/log"
-
-	istioclientset "github.com/knative/pkg/client/clientset/versioned"
-	istioinformers "github.com/knative/pkg/client/informers/externalversions"
-	applicationclientset "github.com/kubernetes-sigs/application/pkg/client/clientset/versioned"
-	applicationinformers "github.com/kubernetes-sigs/application/pkg/client/informers/externalversions"
-	s2iclientset "github.com/kubesphere/s2ioperator/pkg/client/clientset/versioned"
-	s2iinformers "github.com/kubesphere/s2ioperator/pkg/client/informers/externalversions"
-	kubesphereclientset "kubesphere.io/kubesphere/pkg/client/clientset/versioned"
-	kubesphereinformers "kubesphere.io/kubesphere/pkg/client/informers/externalversions"
+	"kubesphere.io/kubesphere/pkg/informers"
+	"kubesphere.io/kubesphere/pkg/simple/client/devops"
+	"kubesphere.io/kubesphere/pkg/simple/client/k8s"
+	ldapclient "kubesphere.io/kubesphere/pkg/simple/client/ldap"
+	"kubesphere.io/kubesphere/pkg/simple/client/multicluster"
+	"kubesphere.io/kubesphere/pkg/simple/client/network"
+	ippoolclient "kubesphere.io/kubesphere/pkg/simple/client/network/ippool"
+	"kubesphere.io/kubesphere/pkg/simple/client/s3"
 )
 
-const defaultResync = 600 * time.Second
+func addControllers(
+	mgr manager.Manager,
+	client k8s.Client,
+	informerFactory informers.InformerFactory,
+	devopsClient devops.Interface,
+	s3Client s3.Interface,
+	ldapClient ldapclient.Interface,
+	options *k8s.KubernetesOptions,
+	authenticationOptions *authoptions.AuthenticationOptions,
+	multiClusterOptions *multicluster.Options,
+	networkOptions *network.Options,
+	serviceMeshEnabled bool,
+	kubectlImage string,
+	stopCh <-chan struct{}) error {
 
-var log = logf.Log.WithName("controller-manager")
+	kubernetesInformer := informerFactory.KubernetesSharedInformerFactory()
+	istioInformer := informerFactory.IstioSharedInformerFactory()
+	kubesphereInformer := informerFactory.KubeSphereSharedInformerFactory()
 
-func AddControllers(mgr manager.Manager, cfg *rest.Config, stopCh <-chan struct{}) error {
+	multiClusterEnabled := multiClusterOptions.Enable
 
-	kubeClient, err := kubernetes.NewForConfig(cfg)
-	if err != nil {
-		log.Error(err, "building kubernetes client failed")
+	var vsController, drController manager.Runnable
+	if serviceMeshEnabled {
+		vsController = virtualservice.NewVirtualServiceController(kubernetesInformer.Core().V1().Services(),
+			istioInformer.Networking().V1alpha3().VirtualServices(),
+			istioInformer.Networking().V1alpha3().DestinationRules(),
+			kubesphereInformer.Servicemesh().V1alpha2().Strategies(),
+			client.Kubernetes(),
+			client.Istio(),
+			client.KubeSphere())
+
+		drController = destinationrule.NewDestinationRuleController(kubernetesInformer.Apps().V1().Deployments(),
+			istioInformer.Networking().V1alpha3().DestinationRules(),
+			kubernetesInformer.Core().V1().Services(),
+			kubesphereInformer.Servicemesh().V1alpha2().ServicePolicies(),
+			client.Kubernetes(),
+			client.Istio(),
+			client.KubeSphere())
 	}
 
-	istioclient, err := istioclientset.NewForConfig(cfg)
-	if err != nil {
-		log.Error(err, "create istio client failed")
-		return err
+	jobController := job.NewJobController(kubernetesInformer.Batch().V1().Jobs(), client.Kubernetes())
+
+	var s2iBinaryController, s2iRunController, devopsProjectController, devopsPipelineController, devopsCredentialController manager.Runnable
+	if devopsClient != nil {
+		s2iBinaryController = s2ibinary.NewController(client.Kubernetes(),
+			client.KubeSphere(),
+			kubesphereInformer.Devops().V1alpha1().S2iBinaries(),
+			s3Client,
+		)
+
+		s2iRunController = s2irun.NewS2iRunController(client.Kubernetes(),
+			client.KubeSphere(),
+			kubesphereInformer.Devops().V1alpha1().S2iBinaries(),
+			kubesphereInformer.Devops().V1alpha1().S2iRuns())
+
+		devopsProjectController = devopsproject.NewController(client.Kubernetes(),
+			client.KubeSphere(), devopsClient,
+			informerFactory.KubernetesSharedInformerFactory().Core().V1().Namespaces(),
+			informerFactory.KubeSphereSharedInformerFactory().Devops().V1alpha3().DevOpsProjects(),
+			informerFactory.KubeSphereSharedInformerFactory().Tenant().V1alpha1().Workspaces())
+
+		devopsPipelineController = pipeline.NewController(client.Kubernetes(),
+			client.KubeSphere(),
+			devopsClient,
+			informerFactory.KubernetesSharedInformerFactory().Core().V1().Namespaces(),
+			informerFactory.KubeSphereSharedInformerFactory().Devops().V1alpha3().Pipelines())
+
+		devopsCredentialController = devopscredential.NewController(client.Kubernetes(),
+			devopsClient,
+			informerFactory.KubernetesSharedInformerFactory().Core().V1().Namespaces(),
+			informerFactory.KubernetesSharedInformerFactory().Core().V1().Secrets())
+
 	}
 
-	applicationClient, err := applicationclientset.NewForConfig(cfg)
-	if err != nil {
-		log.Error(err, "create application client failed")
-		return err
-	}
-	s2iclient, err := s2iclientset.NewForConfig(cfg)
-	if err != nil {
-		log.Error(err, "create s2i client failed")
-		return err
-	}
-	kubesphereclient, err := kubesphereclientset.NewForConfig(cfg)
-	if err != nil {
-		log.Error(err, "create kubesphere client failed")
-		return err
-	}
-
-	informerFactory := informers.NewSharedInformerFactory(kubeClient, defaultResync)
-	istioInformer := istioinformers.NewSharedInformerFactory(istioclient, defaultResync)
-	applicationInformer := applicationinformers.NewSharedInformerFactory(applicationClient, defaultResync)
-	s2iInformer := s2iinformers.NewSharedInformerFactory(s2iclient, defaultResync)
-
-	kubesphereInformer := kubesphereinformers.NewSharedInformerFactory(kubesphereclient, defaultResync)
-
-	vsController := virtualservice.NewVirtualServiceController(informerFactory.Core().V1().Services(),
-		istioInformer.Networking().V1alpha3().VirtualServices(),
-		istioInformer.Networking().V1alpha3().DestinationRules(),
-		kubesphereInformer.Servicemesh().V1alpha2().Strategies(),
-		kubeClient,
-		istioclient,
-		kubesphereclient)
-
-	drController := destinationrule.NewDestinationRuleController(informerFactory.Apps().V1().Deployments(),
-		istioInformer.Networking().V1alpha3().DestinationRules(),
-		informerFactory.Core().V1().Services(),
-		kubesphereInformer.Servicemesh().V1alpha2().ServicePolicies(),
-		kubeClient,
-		istioclient,
-		kubesphereclient)
-
-	apController := application.NewApplicationController(informerFactory.Core().V1().Services(),
-		informerFactory.Apps().V1().Deployments(),
-		informerFactory.Apps().V1().StatefulSets(),
-		kubesphereInformer.Servicemesh().V1alpha2().Strategies(),
-		kubesphereInformer.Servicemesh().V1alpha2().ServicePolicies(),
-		applicationInformer.App().V1beta1().Applications(),
-		kubeClient,
-		applicationClient)
-
-	jobController := job.NewJobController(informerFactory.Batch().V1().Jobs(), kubeClient)
-
-	s2iBinaryController := s2ibinary.NewController(kubesphereclient,
-		kubeClient,
-		kubesphereInformer.Devops().V1alpha1().S2iBinaries())
-
-	s2iRunController := s2irun.NewController(kubesphereclient, s2iclient, kubeClient,
-		kubesphereInformer.Devops().V1alpha1().S2iBinaries(),
-		s2iInformer.Devops().V1alpha1().S2iRuns())
+	storageCapabilityController := capability.NewController(
+		client.KubeSphere().StorageV1alpha1().StorageClassCapabilities(),
+		kubesphereInformer.Storage().V1alpha1(),
+		client.Kubernetes().StorageV1().StorageClasses(),
+		kubernetesInformer.Storage().V1().StorageClasses(),
+		capability.SnapshotSupported(client.Kubernetes().Discovery()),
+		client.Snapshot().SnapshotV1beta1().VolumeSnapshotClasses(),
+		informerFactory.SnapshotSharedInformerFactory().Snapshot().V1beta1().VolumeSnapshotClasses(),
+		kubernetesInformer.Storage().V1beta1().CSIDrivers(),
+	)
 
 	volumeExpansionController := expansion.NewVolumeExpansionController(
-		kubeClient,
-		informerFactory.Core().V1().PersistentVolumeClaims(),
-		informerFactory.Storage().V1().StorageClasses(),
-		informerFactory.Core().V1().Pods(),
-		informerFactory.Apps().V1().Deployments(),
-		informerFactory.Apps().V1().ReplicaSets(),
-		informerFactory.Apps().V1().StatefulSets())
+		client.Kubernetes(),
+		kubernetesInformer.Core().V1().PersistentVolumeClaims(),
+		kubernetesInformer.Storage().V1().StorageClasses(),
+		kubernetesInformer.Core().V1().Pods(),
+		kubernetesInformer.Apps().V1().Deployments(),
+		kubernetesInformer.Apps().V1().ReplicaSets(),
+		kubernetesInformer.Apps().V1().StatefulSets())
 
-	kubesphereInformer.Start(stopCh)
-	istioInformer.Start(stopCh)
-	informerFactory.Start(stopCh)
-	applicationInformer.Start(stopCh)
-	s2iInformer.Start(stopCh)
+	var fedUserCache, fedGlobalRoleBindingCache, fedGlobalRoleCache cache.Store
+	var fedUserCacheController, fedGlobalRoleBindingCacheController, fedGlobalRoleCacheController cache.Controller
+
+	if multiClusterEnabled {
+		fedUserClient, err := util.NewResourceClient(client.Config(), &iamv1alpha2.FedUserResource)
+		if err != nil {
+			klog.Error(err)
+			return err
+		}
+		fedGlobalRoleClient, err := util.NewResourceClient(client.Config(), &iamv1alpha2.FedGlobalRoleResource)
+		if err != nil {
+			klog.Error(err)
+			return err
+		}
+		fedGlobalRoleBindingClient, err := util.NewResourceClient(client.Config(), &iamv1alpha2.FedGlobalRoleBindingResource)
+		if err != nil {
+			klog.Error(err)
+			return err
+		}
+
+		fedUserCache, fedUserCacheController = util.NewResourceInformer(fedUserClient, "", &iamv1alpha2.FedUserResource, func(object runtime.Object) {})
+		fedGlobalRoleCache, fedGlobalRoleCacheController = util.NewResourceInformer(fedGlobalRoleClient, "", &iamv1alpha2.FedGlobalRoleResource, func(object runtime.Object) {})
+		fedGlobalRoleBindingCache, fedGlobalRoleBindingCacheController = util.NewResourceInformer(fedGlobalRoleBindingClient, "", &iamv1alpha2.FedGlobalRoleBindingResource, func(object runtime.Object) {})
+
+		go fedUserCacheController.Run(stopCh)
+		go fedGlobalRoleCacheController.Run(stopCh)
+		go fedGlobalRoleBindingCacheController.Run(stopCh)
+	}
+
+	userController := user.NewUserController(client.Kubernetes(), client.KubeSphere(), client.Config(),
+		kubesphereInformer.Iam().V1alpha2().Users(),
+		kubesphereInformer.Iam().V1alpha2().LoginRecords(),
+		fedUserCache, fedUserCacheController,
+		kubernetesInformer.Core().V1().ConfigMaps(),
+		ldapClient, devopsClient,
+		authenticationOptions, multiClusterEnabled)
+
+	loginRecordController := loginrecord.NewLoginRecordController(
+		client.Kubernetes(),
+		client.KubeSphere(),
+		kubesphereInformer.Iam().V1alpha2().LoginRecords(),
+		kubesphereInformer.Iam().V1alpha2().Users(),
+		authenticationOptions.LoginHistoryRetentionPeriod,
+		authenticationOptions.LoginHistoryMaximumEntries)
+
+	csrController := certificatesigningrequest.NewController(client.Kubernetes(),
+		kubernetesInformer.Certificates().V1beta1().CertificateSigningRequests(),
+		kubernetesInformer.Core().V1().ConfigMaps(), client.Config())
+
+	clusterRoleBindingController := clusterrolebinding.NewController(client.Kubernetes(),
+		kubernetesInformer.Rbac().V1().ClusterRoleBindings(),
+		kubernetesInformer.Apps().V1().Deployments(),
+		kubernetesInformer.Core().V1().Pods(),
+		kubesphereInformer.Iam().V1alpha2().Users(),
+		kubectlImage)
+
+	globalRoleController := globalrole.NewController(client.Kubernetes(), client.KubeSphere(),
+		kubesphereInformer.Iam().V1alpha2().GlobalRoles(), fedGlobalRoleCache, fedGlobalRoleCacheController)
+
+	globalRoleBindingController := globalrolebinding.NewController(client.Kubernetes(), client.KubeSphere(),
+		kubesphereInformer.Iam().V1alpha2().GlobalRoleBindings(),
+		fedGlobalRoleBindingCache, fedGlobalRoleBindingCacheController,
+		multiClusterEnabled)
+
+	groupBindingController := groupbinding.NewController(client.Kubernetes(), client.KubeSphere(),
+		kubesphereInformer.Iam().V1alpha2().GroupBindings(),
+		kubesphereInformer.Types().V1beta1().FederatedGroupBindings(),
+		multiClusterEnabled)
+
+	groupController := group.NewController(client.Kubernetes(), client.KubeSphere(),
+		kubesphereInformer.Iam().V1alpha2().Groups(),
+		kubesphereInformer.Types().V1beta1().FederatedGroups(),
+		multiClusterEnabled)
+
+	var clusterController manager.Runnable
+	if multiClusterEnabled {
+		clusterController = cluster.NewClusterController(
+			client.Kubernetes(),
+			client.Config(),
+			kubesphereInformer.Cluster().V1alpha1().Clusters(),
+			client.KubeSphere().ClusterV1alpha1().Clusters(),
+			multiClusterOptions.ClusterControllerResyncSecond)
+	}
+
+	var nsnpController manager.Runnable
+	if networkOptions.EnableNetworkPolicy {
+		nsnpProvider, err := provider.NewNsNetworkPolicyProvider(client.Kubernetes(), kubernetesInformer.Networking().V1().NetworkPolicies())
+		if err != nil {
+			return err
+		}
+
+		nsnpController = nsnetworkpolicy.NewNSNetworkPolicyController(client.Kubernetes(),
+			client.KubeSphere().NetworkV1alpha1(),
+			kubesphereInformer.Network().V1alpha1().NamespaceNetworkPolicies(),
+			kubernetesInformer.Core().V1().Services(),
+			kubernetesInformer.Core().V1().Nodes(),
+			kubesphereInformer.Tenant().V1alpha1().Workspaces(),
+			kubernetesInformer.Core().V1().Namespaces(), nsnpProvider, networkOptions.NSNPOptions)
+	}
+
+	var ippoolController manager.Runnable
+	ippoolProvider := ippoolclient.NewProvider(kubernetesInformer, client.KubeSphere(), client.Kubernetes(), networkOptions.IPPoolType, options)
+	if ippoolProvider != nil {
+		ippoolController = ippool.NewIPPoolController(kubesphereInformer, kubernetesInformer, client.Kubernetes(), client.KubeSphere(), ippoolProvider)
+	}
 
 	controllers := map[string]manager.Runnable{
-		"virtualservice-controller":  vsController,
-		"destinationrule-controller": drController,
-		"application-controller":     apController,
-		"job-controller":             jobController,
-		"s2ibinary-controller":       s2iBinaryController,
-		"s2irun-controller":          s2iRunController,
-		"volumeexpansion-controller": volumeExpansionController,
+		"virtualservice-controller":     vsController,
+		"destinationrule-controller":    drController,
+		"job-controller":                jobController,
+		"s2ibinary-controller":          s2iBinaryController,
+		"s2irun-controller":             s2iRunController,
+		"storagecapability-controller":  storageCapabilityController,
+		"volumeexpansion-controller":    volumeExpansionController,
+		"user-controller":               userController,
+		"loginrecord-controller":        loginRecordController,
+		"cluster-controller":            clusterController,
+		"nsnp-controller":               nsnpController,
+		"csr-controller":                csrController,
+		"clusterrolebinding-controller": clusterRoleBindingController,
+		"globalrolebinding-controller":  globalRoleBindingController,
+		"ippool-controller":             ippoolController,
+		"groupbinding-controller":       groupBindingController,
+		"group-controller":              groupController,
+	}
+
+	if devopsClient != nil {
+		controllers["pipeline-controller"] = devopsPipelineController
+		controllers["devopsprojects-controller"] = devopsProjectController
+		controllers["devopscredential-controller"] = devopsCredentialController
+	}
+
+	if multiClusterEnabled {
+		controllers["globalrole-controller"] = globalRoleController
+		notificationController, err := notification.NewController(client.Kubernetes(), mgr.GetClient(), mgr.GetCache())
+		if err != nil {
+			return err
+		}
+		controllers["notification-controller"] = notificationController
 	}
 
 	for name, ctrl := range controllers {
-		err = mgr.Add(ctrl)
-		if err != nil {
-			log.Error(err, "add controller to manager failed", "name", name)
+		if ctrl == nil {
+			klog.V(4).Infof("%s is not going to run due to dependent component disabled.", name)
+			continue
+		}
+
+		if err := mgr.Add(ctrl); err != nil {
+			klog.Error(err, "add controller to manager failed", "name", name)
 			return err
 		}
 	}

cmd/controller-manager/app/helper.go

@@ -1,21 +1,39 @@
+/*
+Copyright 2020 KubeSphere Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
 package app
 
 import (
+	"context"
 	"fmt"
+	"net/http"
+	"time"
+
 	"k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
 	"k8s.io/apimachinery/pkg/util/wait"
 	"k8s.io/klog"
-	"net/http"
-	"time"
 )
 
-// WaitForAPIServer waits for the API Server's /healthz endpoint to report "ok" with timeout.
+// WaitForAPIServer waits for the API Server's /healthz endpoint to report "ok" before timeout.
 func WaitForAPIServer(client clientset.Interface, timeout time.Duration) error {
 	var lastErr error
 
 	err := wait.PollImmediate(time.Second, timeout, func() (bool, error) {
 		healthStatus := 0
-		result := client.Discovery().RESTClient().Get().AbsPath("/healthz").Do().StatusCode(&healthStatus)
+		result := client.Discovery().RESTClient().Get().AbsPath("/healthz").Do(context.Background()).StatusCode(&healthStatus)
 		if result.Error() != nil {
 			lastErr = fmt.Errorf("failed to get apiserver /healthz status: %v", result.Error())
 			return false, nil

cmd/controller-manager/app/options/options.go

@@ -1,65 +1,121 @@
+/*
+Copyright 2020 KubeSphere Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
 package options
 
 import (
 	"flag"
-	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	apiserverconfig "k8s.io/apiserver/pkg/apis/config"
-	cliflag "k8s.io/component-base/cli/flag"
-	"k8s.io/klog"
-	"k8s.io/kubernetes/pkg/client/leaderelectionconfig"
-	kubesphereconfig "kubesphere.io/kubesphere/pkg/server/config"
-	"kubesphere.io/kubesphere/pkg/simple/client/devops"
-	"kubesphere.io/kubesphere/pkg/simple/client/k8s"
-	"kubesphere.io/kubesphere/pkg/simple/client/openpitrix"
-	"kubesphere.io/kubesphere/pkg/simple/client/s2is3"
 	"strings"
 	"time"
+
+	"k8s.io/apimachinery/pkg/labels"
+
+	"github.com/spf13/pflag"
+	"k8s.io/client-go/tools/leaderelection"
+	cliflag "k8s.io/component-base/cli/flag"
+	"k8s.io/klog"
+
+	authoptions "kubesphere.io/kubesphere/pkg/apiserver/authentication/options"
+	"kubesphere.io/kubesphere/pkg/simple/client/devops/jenkins"
+	"kubesphere.io/kubesphere/pkg/simple/client/k8s"
+	ldapclient "kubesphere.io/kubesphere/pkg/simple/client/ldap"
+	"kubesphere.io/kubesphere/pkg/simple/client/multicluster"
+	"kubesphere.io/kubesphere/pkg/simple/client/network"
+	"kubesphere.io/kubesphere/pkg/simple/client/openpitrix"
+	"kubesphere.io/kubesphere/pkg/simple/client/s3"
+	"kubesphere.io/kubesphere/pkg/simple/client/servicemesh"
 )
 
 type KubeSphereControllerManagerOptions struct {
-	KubernetesOptions *k8s.KubernetesOptions
-	DevopsOptions     *devops.DevopsOptions
-	S3Options         *s2is3.S3Options
-	OpenPitrixOptions *openpitrix.OpenPitrixOptions
+	KubernetesOptions     *k8s.KubernetesOptions
+	DevopsOptions         *jenkins.Options
+	S3Options             *s3.Options
+	AuthenticationOptions *authoptions.AuthenticationOptions
+	LdapOptions           *ldapclient.Options
+	OpenPitrixOptions     *openpitrix.Options
+	NetworkOptions        *network.Options
+	MultiClusterOptions   *multicluster.Options
+	ServiceMeshOptions    *servicemesh.Options
+	LeaderElect           bool
+	LeaderElection        *leaderelection.LeaderElectionConfig
+	WebhookCertDir        string
 
-	LeaderElection *apiserverconfig.LeaderElectionConfiguration
+	// KubeSphere is using sigs.k8s.io/application as fundamental object to implement Application Management.
+	// There are other projects also built on sigs.k8s.io/application, when KubeSphere installed along side
+	// them, conflicts happen. So we leave an option to only reconcile applications  matched with the given
+	// selector. Default will reconcile all applications.
+	//    For example
+	//      "kubesphere.io/creator=" means reconcile applications with this label key
+	//      "!kubesphere.io/creator" means exclude applications with this key
+	ApplicationSelector string
 }
 
 func NewKubeSphereControllerManagerOptions() *KubeSphereControllerManagerOptions {
 	s := &KubeSphereControllerManagerOptions{
-		KubernetesOptions: k8s.NewKubernetesOptions(),
-		DevopsOptions:     devops.NewDevopsOptions(),
-		S3Options:         s2is3.NewS3Options(),
-		OpenPitrixOptions: openpitrix.NewOpenPitrixOptions(),
-		LeaderElection: &apiserverconfig.LeaderElectionConfiguration{
-			LeaderElect:   false,
-			LeaseDuration: v1.Duration{Duration: 30 * time.Second},
-			RenewDeadline: v1.Duration{Duration: 15 * time.Second},
-			RetryPeriod:   v1.Duration{Duration: 5 * time.Second},
-			ResourceLock:  "ks-controller-manager-leader-election",
+		KubernetesOptions:     k8s.NewKubernetesOptions(),
+		DevopsOptions:         jenkins.NewDevopsOptions(),
+		S3Options:             s3.NewS3Options(),
+		LdapOptions:           ldapclient.NewOptions(),
+		OpenPitrixOptions:     openpitrix.NewOptions(),
+		NetworkOptions:        network.NewNetworkOptions(),
+		MultiClusterOptions:   multicluster.NewOptions(),
+		ServiceMeshOptions:    servicemesh.NewServiceMeshOptions(),
+		AuthenticationOptions: authoptions.NewAuthenticateOptions(),
+		LeaderElection: &leaderelection.LeaderElectionConfig{
+			LeaseDuration: 30 * time.Second,
+			RenewDeadline: 15 * time.Second,
+			RetryPeriod:   5 * time.Second,
 		},
+		LeaderElect:         false,
+		WebhookCertDir:      "",
+		ApplicationSelector: "",
 	}
 
 	return s
 }
 
-func (s *KubeSphereControllerManagerOptions) ApplyTo(conf *kubesphereconfig.Config) {
-	s.S3Options.ApplyTo(conf.S3Options)
-	s.KubernetesOptions.ApplyTo(conf.KubernetesOptions)
-	s.DevopsOptions.ApplyTo(conf.DevopsOptions)
-	s.OpenPitrixOptions.ApplyTo(conf.OpenPitrixOptions)
-}
-
 func (s *KubeSphereControllerManagerOptions) Flags() cliflag.NamedFlagSets {
 	fss := cliflag.NamedFlagSets{}
 
-	s.KubernetesOptions.AddFlags(fss.FlagSet("kubernetes"))
-	s.DevopsOptions.AddFlags(fss.FlagSet("devops"))
-	s.S3Options.AddFlags(fss.FlagSet("s3"))
-	s.OpenPitrixOptions.AddFlags(fss.FlagSet("openpitrix"))
+	s.KubernetesOptions.AddFlags(fss.FlagSet("kubernetes"), s.KubernetesOptions)
+	s.DevopsOptions.AddFlags(fss.FlagSet("devops"), s.DevopsOptions)
+	s.S3Options.AddFlags(fss.FlagSet("s3"), s.S3Options)
+	s.AuthenticationOptions.AddFlags(fss.FlagSet("authentication"), s.AuthenticationOptions)
+	s.LdapOptions.AddFlags(fss.FlagSet("ldap"), s.LdapOptions)
+	s.OpenPitrixOptions.AddFlags(fss.FlagSet("openpitrix"), s.OpenPitrixOptions)
+	s.NetworkOptions.AddFlags(fss.FlagSet("network"), s.NetworkOptions)
+	s.MultiClusterOptions.AddFlags(fss.FlagSet("multicluster"), s.MultiClusterOptions)
+	s.ServiceMeshOptions.AddFlags(fss.FlagSet("servicemesh"), s.ServiceMeshOptions)
 
 	fs := fss.FlagSet("leaderelection")
-	leaderelectionconfig.BindFlags(s.LeaderElection, fs)
+	s.bindLeaderElectionFlags(s.LeaderElection, fs)
+
+	fs.BoolVar(&s.LeaderElect, "leader-elect", s.LeaderElect, ""+
+		"Whether to enable leader election. This field should be enabled when controller manager"+
+		"deployed with multiple replicas.")
+
+	fs.StringVar(&s.WebhookCertDir, "webhook-cert-dir", s.WebhookCertDir, ""+
+		"Certificate directory used to setup webhooks, need tls.crt and tls.key placed inside."+
+		"if not set, webhook server would look up the server key and certificate in"+
+		"{TempDir}/k8s-webhook-server/serving-certs")
+
+	gfs := fss.FlagSet("generic")
+	gfs.StringVar(&s.ApplicationSelector, "application-selector", s.ApplicationSelector, ""+
+		"Only reconcile application(sigs.k8s.io/application) objects match given selector, this could avoid conflicts with "+
+		"other projects built on top of sig-application. Default behavior is to reconcile all of application objects.")
 
 	kfs := fss.FlagSet("klog")
 	local := flag.NewFlagSet("klog", flag.ExitOnError)
@@ -78,5 +134,31 @@ func (s *KubeSphereControllerManagerOptions) Validate() []error {
 	errs = append(errs, s.KubernetesOptions.Validate()...)
 	errs = append(errs, s.S3Options.Validate()...)
 	errs = append(errs, s.OpenPitrixOptions.Validate()...)
+	errs = append(errs, s.NetworkOptions.Validate()...)
+	errs = append(errs, s.LdapOptions.Validate()...)
+
+	if len(s.ApplicationSelector) != 0 {
+		_, err := labels.Parse(s.ApplicationSelector)
+		if err != nil {
+			errs = append(errs, err)
+		}
+	}
+
 	return errs
 }
+
+func (s *KubeSphereControllerManagerOptions) bindLeaderElectionFlags(l *leaderelection.LeaderElectionConfig, fs *pflag.FlagSet) {
+	fs.DurationVar(&l.LeaseDuration, "leader-elect-lease-duration", l.LeaseDuration, ""+
+		"The duration that non-leader candidates will wait after observing a leadership "+
+		"renewal until attempting to acquire leadership of a led but unrenewed leader "+
+		"slot. This is effectively the maximum duration that a leader can be stopped "+
+		"before it is replaced by another candidate. This is only applicable if leader "+
+		"election is enabled.")
+	fs.DurationVar(&l.RenewDeadline, "leader-elect-renew-deadline", l.RenewDeadline, ""+
+		"The interval between attempts by the acting master to renew a leadership slot "+
+		"before it stops leading. This must be less than or equal to the lease duration. "+
+		"This is only applicable if leader election is enabled.")
+	fs.DurationVar(&l.RetryPeriod, "leader-elect-retry-period", l.RetryPeriod, ""+
+		"The duration the clients should wait between attempting acquisition and renewal "+
+		"of a leadership. This is only applicable if leader election is enabled.")
+}

cmd/controller-manager/app/server.go

@@ -1,72 +1,103 @@
 /*
+Copyright 2019 The KubeSphere Authors.
 
- Copyright 2019 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
 
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
-     http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
+    http://www.apache.org/licenses/LICENSE-2.0
 
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
 */
 
 package app
 
 import (
-	"context"
 	"fmt"
+	"os"
+
 	"github.com/spf13/cobra"
-	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
 	utilerrors "k8s.io/apimachinery/pkg/util/errors"
-	"k8s.io/apimachinery/pkg/util/uuid"
-	"k8s.io/client-go/tools/leaderelection"
-	"k8s.io/client-go/tools/leaderelection/resourcelock"
-	"k8s.io/client-go/tools/record"
 	cliflag "k8s.io/component-base/cli/flag"
 	"k8s.io/klog"
-	"kubesphere.io/kubesphere/cmd/controller-manager/app/options"
-	"kubesphere.io/kubesphere/pkg/apis"
-	"kubesphere.io/kubesphere/pkg/client/clientset/versioned/scheme"
-	"kubesphere.io/kubesphere/pkg/controller"
-	controllerconfig "kubesphere.io/kubesphere/pkg/server/config"
-	"kubesphere.io/kubesphere/pkg/simple/client"
-	"kubesphere.io/kubesphere/pkg/utils/term"
-	"os"
+	"k8s.io/klog/klogr"
+	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/manager"
 	"sigs.k8s.io/controller-runtime/pkg/runtime/signals"
+	"sigs.k8s.io/controller-runtime/pkg/webhook"
+
+	"kubesphere.io/kubesphere/cmd/controller-manager/app/options"
+	"kubesphere.io/kubesphere/pkg/apis"
+	controllerconfig "kubesphere.io/kubesphere/pkg/apiserver/config"
+	"kubesphere.io/kubesphere/pkg/controller/application"
+	"kubesphere.io/kubesphere/pkg/controller/namespace"
+	"kubesphere.io/kubesphere/pkg/controller/network/webhooks"
+	"kubesphere.io/kubesphere/pkg/controller/openpitrix/helmapplication"
+	"kubesphere.io/kubesphere/pkg/controller/openpitrix/helmcategory"
+	"kubesphere.io/kubesphere/pkg/controller/openpitrix/helmrelease"
+	"kubesphere.io/kubesphere/pkg/controller/openpitrix/helmrepo"
+	"kubesphere.io/kubesphere/pkg/controller/quota"
+	"kubesphere.io/kubesphere/pkg/controller/serviceaccount"
+	"kubesphere.io/kubesphere/pkg/controller/user"
+	"kubesphere.io/kubesphere/pkg/controller/workspace"
+	"kubesphere.io/kubesphere/pkg/controller/workspacerole"
+	"kubesphere.io/kubesphere/pkg/controller/workspacerolebinding"
+	"kubesphere.io/kubesphere/pkg/controller/workspacetemplate"
+	"kubesphere.io/kubesphere/pkg/informers"
+	"kubesphere.io/kubesphere/pkg/simple/client/devops"
+	"kubesphere.io/kubesphere/pkg/simple/client/devops/jenkins"
+	"kubesphere.io/kubesphere/pkg/simple/client/k8s"
+	ldapclient "kubesphere.io/kubesphere/pkg/simple/client/ldap"
+	"kubesphere.io/kubesphere/pkg/simple/client/s3"
+	"kubesphere.io/kubesphere/pkg/utils/metrics"
+	"kubesphere.io/kubesphere/pkg/utils/term"
+	"kubesphere.io/kubesphere/pkg/version"
 )
 
 func NewControllerManagerCommand() *cobra.Command {
 	s := options.NewKubeSphereControllerManagerOptions()
+	conf, err := controllerconfig.TryLoadFromDisk()
+	if err == nil {
+		// make sure LeaderElection is not nil
+		s = &options.KubeSphereControllerManagerOptions{
+			KubernetesOptions:     conf.KubernetesOptions,
+			DevopsOptions:         conf.DevopsOptions,
+			S3Options:             conf.S3Options,
+			AuthenticationOptions: conf.AuthenticationOptions,
+			LdapOptions:           conf.LdapOptions,
+			OpenPitrixOptions:     conf.OpenPitrixOptions,
+			NetworkOptions:        conf.NetworkOptions,
+			MultiClusterOptions:   conf.MultiClusterOptions,
+			ServiceMeshOptions:    conf.ServiceMeshOptions,
+			LeaderElection:        s.LeaderElection,
+			LeaderElect:           s.LeaderElect,
+			WebhookCertDir:        s.WebhookCertDir,
+		}
+	} else {
+		klog.Fatal("Failed to load configuration from disk", err)
+	}
 
 	cmd := &cobra.Command{
 		Use:  "controller-manager",
 		Long: `KubeSphere controller manager is a daemon that`,
 		Run: func(cmd *cobra.Command, args []string) {
-
-			err := controllerconfig.Load()
-			if err != nil {
-				klog.Fatal(err)
-				os.Exit(1)
-			}
-
-			s = Complete(s)
-
 			if errs := s.Validate(); len(errs) != 0 {
 				klog.Error(utilerrors.NewAggregate(errs))
 				os.Exit(1)
 			}
 
-			if err = Run(s, signals.SetupSignalHandler()); err != nil {
+			if err = run(s, signals.SetupSignalHandler()); err != nil {
+				klog.Error(err)
 				os.Exit(1)
 			}
 		},
+		SilenceUsage: true,
 	}
 
 	fs := cmd.Flags()
@@ -79,135 +110,235 @@ func NewControllerManagerCommand() *cobra.Command {
 	usageFmt := "Usage:\n  %s\n"
 	cols, _, _ := term.TerminalSize(cmd.OutOrStdout())
 	cmd.SetHelpFunc(func(cmd *cobra.Command, args []string) {
-		fmt.Fprintf(cmd.OutOrStdout(), "%s\n\n"+usageFmt, cmd.Long, cmd.UseLine())
+		_, _ = fmt.Fprintf(cmd.OutOrStdout(), "%s\n\n"+usageFmt, cmd.Long, cmd.UseLine())
 		cliflag.PrintSections(cmd.OutOrStdout(), namedFlagSets, cols)
 	})
+
+	versionCmd := &cobra.Command{
+		Use:   "version",
+		Short: "Print the version of KubeSphere ks-apiserver",
+		Run: func(cmd *cobra.Command, args []string) {
+			cmd.Println(version.Get())
+		},
+	}
+
+	cmd.AddCommand(versionCmd)
+
 	return cmd
 }
 
-func Complete(s *options.KubeSphereControllerManagerOptions) *options.KubeSphereControllerManagerOptions {
-	conf := controllerconfig.Get()
+func run(s *options.KubeSphereControllerManagerOptions, stopCh <-chan struct{}) error {
 
-	conf.Apply(&controllerconfig.Config{
-		DevopsOptions:     s.DevopsOptions,
-		KubernetesOptions: s.KubernetesOptions,
-		S3Options:         s.S3Options,
-		OpenPitrixOptions: s.OpenPitrixOptions,
-	})
-
-	out := &options.KubeSphereControllerManagerOptions{
-		KubernetesOptions: conf.KubernetesOptions,
-		DevopsOptions:     conf.DevopsOptions,
-		S3Options:         conf.S3Options,
-		OpenPitrixOptions: conf.OpenPitrixOptions,
-		LeaderElection:    s.LeaderElection,
-	}
-
-	return out
-}
-
-func CreateClientSet(conf *controllerconfig.Config, stopCh <-chan struct{}) error {
-	csop := &client.ClientSetOptions{}
-
-	csop.SetKubernetesOptions(conf.KubernetesOptions).
-		SetDevopsOptions(conf.DevopsOptions).
-		SetS3Options(conf.S3Options).
-		SetOpenPitrixOptions(conf.OpenPitrixOptions).
-		SetKubeSphereOptions(conf.KubeSphereOptions)
-	client.NewClientSetFactory(csop, stopCh)
-
-	return nil
-}
-
-func Run(s *options.KubeSphereControllerManagerOptions, stopCh <-chan struct{}) error {
-	err := CreateClientSet(controllerconfig.Get(), stopCh)
+	kubernetesClient, err := k8s.NewKubernetesClient(s.KubernetesOptions)
 	if err != nil {
-		klog.Error(err)
+		klog.Errorf("Failed to create kubernetes clientset %v", err)
 		return err
 	}
 
-	config := client.ClientSets().K8s().Config()
-
-	run := func(ctx context.Context) {
-		klog.V(0).Info("setting up manager")
-		mgr, err := manager.New(config, manager.Options{})
+	var devopsClient devops.Interface
+	if s.DevopsOptions != nil && len(s.DevopsOptions.Host) != 0 {
+		devopsClient, err = jenkins.NewDevopsClient(s.DevopsOptions)
 		if err != nil {
-			klog.Fatalf("unable to set up overall controller manager: %v", err)
+			return fmt.Errorf("failed to connect jenkins, please check jenkins status, error: %v", err)
 		}
-
-		klog.V(0).Info("setting up scheme")
-		if err := apis.AddToScheme(mgr.GetScheme()); err != nil {
-			klog.Fatalf("unable add APIs to scheme: %v", err)
-		}
-
-		klog.V(0).Info("Setting up controllers")
-		if err := controller.AddToManager(mgr); err != nil {
-			klog.Fatalf("unable to register controllers to the manager: %v", err)
-		}
-
-		if err := AddControllers(mgr, config, stopCh); err != nil {
-			klog.Fatalf("unable to register controllers to the manager: %v", err)
-		}
-
-		klog.V(0).Info("Starting the Cmd.")
-		if err := mgr.Start(stopCh); err != nil {
-			klog.Fatalf("unable to run the manager: %v", err)
-		}
-
-		select {}
 	}
 
-	if !s.LeaderElection.LeaderElect {
-		run(context.TODO())
+	var ldapClient ldapclient.Interface
+	// when there is no ldapOption, we set ldapClient as nil, which means we don't need to sync user info into ldap.
+	if s.LdapOptions != nil && len(s.LdapOptions.Host) != 0 {
+		if s.LdapOptions.Host == ldapclient.FAKE_HOST { // for debug only
+			ldapClient = ldapclient.NewSimpleLdap()
+		} else {
+			ldapClient, err = ldapclient.NewLdapClient(s.LdapOptions, stopCh)
+			if err != nil {
+				return fmt.Errorf("failed to connect to ldap service, please check ldap status, error: %v", err)
+			}
+		}
+	} else {
+		klog.Warning("ks-controller-manager starts without ldap provided, it will not sync user into ldap")
 	}
 
-	ctx, cancel := context.WithCancel(context.Background())
-	defer cancel()
+	var s3Client s3.Interface
+	if s.S3Options != nil && len(s.S3Options.Endpoint) != 0 {
+		s3Client, err = s3.NewS3Client(s.S3Options)
+		if err != nil {
+			return fmt.Errorf("failed to connect to s3, please check s3 service status, error: %v", err)
+		}
+	}
 
-	go func() {
-		<-stopCh
-		cancel()
-	}()
+	informerFactory := informers.NewInformerFactories(
+		kubernetesClient.Kubernetes(),
+		kubernetesClient.KubeSphere(),
+		kubernetesClient.Istio(),
+		kubernetesClient.Snapshot(),
+		kubernetesClient.ApiExtensions(),
+		kubernetesClient.Prometheus())
 
-	id, err := os.Hostname()
+	mgrOptions := manager.Options{
+		CertDir: s.WebhookCertDir,
+		Port:    8443,
+	}
+
+	if s.LeaderElect {
+		mgrOptions = manager.Options{
+			CertDir:                 s.WebhookCertDir,
+			Port:                    8443,
+			LeaderElection:          s.LeaderElect,
+			LeaderElectionNamespace: "kubesphere-system",
+			LeaderElectionID:        "ks-controller-manager-leader-election",
+			LeaseDuration:           &s.LeaderElection.LeaseDuration,
+			RetryPeriod:             &s.LeaderElection.RetryPeriod,
+			RenewDeadline:           &s.LeaderElection.RenewDeadline,
+		}
+	}
+
+	klog.V(0).Info("setting up manager")
+	ctrl.SetLogger(klogr.New())
+	// Use 8443 instead of 443 cause we need root permission to bind port 443
+	mgr, err := manager.New(kubernetesClient.Config(), mgrOptions)
 	if err != nil {
-		return err
+		klog.Fatalf("unable to set up overall controller manager: %v", err)
 	}
 
-	// add a uniquifier so that two processes on the same host don't accidentally both become active
-	id = id + "_" + string(uuid.NewUUID())
+	if err = apis.AddToScheme(mgr.GetScheme()); err != nil {
+		klog.Fatalf("unable add APIs to scheme: %v", err)
+	}
 
-	// TODO: change lockType to lease
-	// once we finished moving to Kubernetes v1.16+, we
-	// change lockType to lease
-	lock, err := resourcelock.New("endpoints",
-		"kubesphere-system",
-		s.LeaderElection.ResourceLock,
-		client.ClientSets().K8s().Kubernetes().CoreV1(),
-		resourcelock.ResourceLockConfig{
-			Identity: id,
-			EventRecorder: record.NewBroadcaster().NewRecorder(scheme.Scheme, v1.EventSource{
-				Component: "ks-controller-manager",
-			}),
-		})
+	// register common meta types into schemas.
+	metav1.AddToGroupVersion(mgr.GetScheme(), metav1.SchemeGroupVersion)
+
+	workspaceTemplateReconciler := &workspacetemplate.Reconciler{MultiClusterEnabled: s.MultiClusterOptions.Enable}
+	if err = workspaceTemplateReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create workspace template controller: %v", err)
+	}
+
+	workspaceReconciler := &workspace.Reconciler{}
+	if err = workspaceReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create workspace controller: %v", err)
+	}
+
+	workspaceRoleReconciler := &workspacerole.Reconciler{MultiClusterEnabled: s.MultiClusterOptions.Enable}
+	if err = workspaceRoleReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create workspace role controller: %v", err)
+	}
+
+	workspaceRoleBindingReconciler := &workspacerolebinding.Reconciler{MultiClusterEnabled: s.MultiClusterOptions.Enable}
+	if err = workspaceRoleBindingReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create workspace role binding controller: %v", err)
+	}
+
+	namespaceReconciler := &namespace.Reconciler{}
+	if err = namespaceReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create namespace controller: %v", err)
+	}
+
+	err = helmrepo.Add(mgr)
+	if err != nil {
+		klog.Fatal("Unable to create helm repo controller")
+	}
+
+	err = helmcategory.Add(mgr)
+	if err != nil {
+		klog.Fatal("Unable to create helm category controller")
+	}
+
+	var opS3Client s3.Interface
+	if !s.OpenPitrixOptions.AppStoreConfIsEmpty() {
+		opS3Client, err = s3.NewS3Client(s.OpenPitrixOptions.S3Options)
+		if err != nil {
+			klog.Fatalf("failed to connect to s3, please check openpitrix s3 service status, error: %v", err)
+		}
+		err = (&helmapplication.ReconcileHelmApplication{}).SetupWithManager(mgr)
+		if err != nil {
+			klog.Fatalf("Unable to create helm application controller, error: %s", err)
+		}
+
+		err = (&helmapplication.ReconcileHelmApplicationVersion{}).SetupWithManager(mgr)
+		if err != nil {
+			klog.Fatalf("Unable to create helm application version controller, error: %s ", err)
+		}
+	}
+
+	err = (&helmrelease.ReconcileHelmRelease{
+		// nil interface is valid value.
+		StorageClient:      opS3Client,
+		KsFactory:          informerFactory.KubeSphereSharedInformerFactory(),
+		MultiClusterEnable: s.MultiClusterOptions.Enable,
+		WaitTime:           s.OpenPitrixOptions.ReleaseControllerOptions.WaitTime,
+		MaxConcurrent:      s.OpenPitrixOptions.ReleaseControllerOptions.MaxConcurrent,
+		StopChan:           stopCh,
+	}).SetupWithManager(mgr)
 
 	if err != nil {
-		klog.Fatalf("error creating lock: %v", err)
+		klog.Fatalf("Unable to create helm release controller, error: %s", err)
 	}
 
-	leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{
-		Lock:          lock,
-		LeaseDuration: s.LeaderElection.LeaseDuration.Duration,
-		RenewDeadline: s.LeaderElection.RenewDeadline.Duration,
-		RetryPeriod:   s.LeaderElection.RetryPeriod.Duration,
-		Callbacks: leaderelection.LeaderCallbacks{
-			OnStartedLeading: run,
-			OnStoppedLeading: func() {
-				klog.Errorf("leadership lost")
-				os.Exit(0)
-			},
-		},
-	})
+	selector, _ := labels.Parse(s.ApplicationSelector)
+	applicationReconciler := &application.ApplicationReconciler{
+		Scheme:              mgr.GetScheme(),
+		Client:              mgr.GetClient(),
+		Mapper:              mgr.GetRESTMapper(),
+		ApplicationSelector: selector,
+	}
+	if err = applicationReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create application controller: %v", err)
+	}
+
+	saReconciler := &serviceaccount.Reconciler{}
+	if err = saReconciler.SetupWithManager(mgr); err != nil {
+		klog.Fatalf("Unable to create ServiceAccount controller: %v", err)
+	}
+
+	resourceQuotaReconciler := quota.Reconciler{}
+	if err := resourceQuotaReconciler.SetupWithManager(mgr, quota.DefaultMaxConcurrentReconciles, quota.DefaultResyncPeriod, informerFactory.KubernetesSharedInformerFactory()); err != nil {
+		klog.Fatalf("Unable to create ResourceQuota controller: %v", err)
+	}
+
+	// TODO(jeff): refactor config with CRD
+	servicemeshEnabled := s.ServiceMeshOptions != nil && len(s.ServiceMeshOptions.IstioPilotHost) != 0
+	if err = addControllers(mgr,
+		kubernetesClient,
+		informerFactory,
+		devopsClient,
+		s3Client,
+		ldapClient,
+		s.KubernetesOptions,
+		s.AuthenticationOptions,
+		s.MultiClusterOptions,
+		s.NetworkOptions,
+		servicemeshEnabled,
+		s.AuthenticationOptions.KubectlImage, stopCh); err != nil {
+		klog.Fatalf("unable to register controllers to the manager: %v", err)
+	}
+
+	// Start cache data after all informer is registered
+	klog.V(0).Info("Starting cache resource from apiserver...")
+	informerFactory.Start(stopCh)
+
+	// Setup webhooks
+	klog.V(2).Info("setting up webhook server")
+	hookServer := mgr.GetWebhookServer()
+
+	klog.V(2).Info("registering webhooks to the webhook server")
+	hookServer.Register("/validate-email-iam-kubesphere-io-v1alpha2", &webhook.Admission{Handler: &user.EmailValidator{Client: mgr.GetClient()}})
+	hookServer.Register("/validate-network-kubesphere-io-v1alpha1", &webhook.Admission{Handler: &webhooks.ValidatingHandler{C: mgr.GetClient()}})
+	hookServer.Register("/mutate-network-kubesphere-io-v1alpha1", &webhook.Admission{Handler: &webhooks.MutatingHandler{C: mgr.GetClient()}})
+
+	resourceQuotaAdmission, err := quota.NewResourceQuotaAdmission(mgr.GetClient(), mgr.GetScheme())
+	if err != nil {
+		klog.Fatalf("unable to create resource quota admission: %v", err)
+	}
+	hookServer.Register("/validate-quota-kubesphere-io-v1alpha2", &webhook.Admission{Handler: resourceQuotaAdmission})
+
+	klog.V(2).Info("registering metrics to the webhook server")
+	// Add an extra metric endpoint, so we can use the the same metric definition with ks-apiserver
+	// /kapis/metrics is independent of controller-manager's built-in /metrics
+	mgr.AddMetricsExtraHandler("/kapis/metrics", metrics.Handler())
+
+	klog.V(0).Info("Starting the controllers.")
+	if err = mgr.Start(stopCh); err != nil {
+		klog.Fatalf("unable to run the manager: %v", err)
+	}
 
 	return nil
 }

cmd/controller-manager/controller-manager.go

@@ -1,8 +1,25 @@
+/*
+Copyright 2020 KubeSphere Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
 package main
 
 import (
-	"kubesphere.io/kubesphere/cmd/controller-manager/app"
 	"os"
+
+	"kubesphere.io/kubesphere/cmd/controller-manager/app"
 )
 
 func main() {

cmd/hypersphere/hypersphere.go

@@ -1,75 +0,0 @@
-package main
-
-import (
-	goflag "flag"
-	cliflag "k8s.io/component-base/cli/flag"
-	"path/filepath"
-
-	"github.com/spf13/cobra"
-	"github.com/spf13/pflag"
-	controllermanager "kubesphere.io/kubesphere/cmd/controller-manager/app"
-	ksapigateway "kubesphere.io/kubesphere/cmd/ks-apigateway/app"
-	ksapiserver "kubesphere.io/kubesphere/cmd/ks-apiserver/app"
-	ksaiam "kubesphere.io/kubesphere/cmd/ks-iam/app"
-	"os"
-)
-
-func main() {
-	hypersphereCommand, allCommandFns := NewHyperSphereCommand()
-
-	pflag.CommandLine.SetNormalizeFunc(cliflag.WordSepNormalizeFunc)
-	pflag.CommandLine.AddGoFlagSet(goflag.CommandLine)
-
-	basename := filepath.Base(os.Args[0])
-	if err := commandFor(basename, hypersphereCommand, allCommandFns).Execute(); err != nil {
-		os.Exit(1)
-	}
-}
-
-func commandFor(basename string, defaultCommand *cobra.Command, commands []func() *cobra.Command) *cobra.Command {
-	for _, commandFn := range commands {
-		command := commandFn()
-		if command.Name() == basename {
-			return command
-		}
-
-		for _, alias := range command.Aliases {
-			if alias == basename {
-				return command
-			}
-		}
-	}
-
-	return defaultCommand
-}
-
-func NewHyperSphereCommand() (*cobra.Command, []func() *cobra.Command) {
-	apiserver := func() *cobra.Command { return ksapiserver.NewAPIServerCommand() }
-	controllermanager := func() *cobra.Command { return controllermanager.NewControllerManagerCommand() }
-	iam := func() *cobra.Command { return ksaiam.NewAPIServerCommand() }
-	apigateway := func() *cobra.Command { return ksapigateway.NewAPIGatewayCommand() }
-
-	commandFns := []func() *cobra.Command{
-		apiserver,
-		controllermanager,
-		iam,
-		apigateway,
-	}
-
-	cmd := &cobra.Command{
-		Use:   "hypersphere",
-		Short: "Request a new project",
-		Run: func(cmd *cobra.Command, args []string) {
-			if len(args) != 0 {
-				cmd.Help()
-				os.Exit(0)
-			}
-		},
-	}
-
-	for i := range commandFns {
-		cmd.AddCommand(commandFns[i]())
-	}
-
-	return cmd, commandFns
-}

cmd/ks-apigateway/apiserver.go

@@ -1,32 +0,0 @@
-/*
-
- Copyright 2019 The KubeSphere Authors.
-
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
-     http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
-*/
-package main
-
-import (
-	"kubesphere.io/kubesphere/cmd/ks-apigateway/app"
-	"os"
-)
-
-func main() {
-
-	cmd := app.NewAPIGatewayCommand()
-
-	if err := cmd.Execute(); err != nil {
-		os.Exit(1)
-	}
-}

cmd/ks-apigateway/app/server.go

@@ -1,53 +0,0 @@
-package app
-
-import (
-	"flag"
-	"github.com/mholt/caddy/caddy/caddymain"
-	"github.com/mholt/caddy/caddyhttp/httpserver"
-	"github.com/spf13/cobra"
-	apiserverconfig "kubesphere.io/kubesphere/pkg/server/config"
-	"kubesphere.io/kubesphere/pkg/simple/client"
-	"kubesphere.io/kubesphere/pkg/utils/signals"
-
-	"kubesphere.io/kubesphere/pkg/apigateway"
-)
-
-func NewAPIGatewayCommand() *cobra.Command {
-
-	cmd := &cobra.Command{
-		Use: "ks-apigateway",
-		Long: `The KubeSphere API Gateway, which is responsible 
-for proxy request to the right backend. API Gateway also proxy 
-Kubernetes API Server for KubeSphere authorization purpose.
-`,
-		RunE: func(cmd *cobra.Command, args []string) error {
-
-			err := apiserverconfig.Load()
-			if err != nil {
-				return err
-			}
-
-			apigateway.RegisterPlugins()
-
-			return Run(signals.SetupSignalHandler())
-		},
-	}
-
-	cmd.Flags().AddGoFlagSet(flag.CommandLine)
-
-	return cmd
-}
-
-func Run(stopCh <-chan struct{}) error {
-
-	csop := &client.ClientSetOptions{}
-	csop.SetKubernetesOptions(apiserverconfig.Get().KubernetesOptions)
-	client.NewClientSetFactory(csop, stopCh)
-
-	httpserver.RegisterDevDirective("authenticate", "jwt")
-	httpserver.RegisterDevDirective("authentication", "jwt")
-	httpserver.RegisterDevDirective("swagger", "jwt")
-	caddymain.Run()
-
-	return nil
-}

cmd/ks-apiserver/apiserver.go

@@ -1,25 +1,25 @@
 /*
+Copyright 2019 The KubeSphere Authors.
 
- Copyright 2019 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
 
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
-     http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
+    http://www.apache.org/licenses/LICENSE-2.0
 
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
 */
+
 package main
 
 import (
-	"kubesphere.io/kubesphere/cmd/ks-apiserver/app"
 	"log"
+
+	"kubesphere.io/kubesphere/cmd/ks-apiserver/app"
 )
 
 func main() {

cmd/ks-apiserver/app/options/options.go

@@ -1,68 +1,94 @@
+/*
+Copyright 2020 KubeSphere Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
 package options
 
 import (
+	"crypto/tls"
 	"flag"
+	"fmt"
+
 	cliflag "k8s.io/component-base/cli/flag"
 	"k8s.io/klog"
+	runtimecache "sigs.k8s.io/controller-runtime/pkg/cache"
+
+	"kubesphere.io/kubesphere/pkg/apis"
+	"kubesphere.io/kubesphere/pkg/apiserver"
+	apiserverconfig "kubesphere.io/kubesphere/pkg/apiserver/config"
+	"kubesphere.io/kubesphere/pkg/client/clientset/versioned/scheme"
+	"kubesphere.io/kubesphere/pkg/informers"
 	genericoptions "kubesphere.io/kubesphere/pkg/server/options"
-	"kubesphere.io/kubesphere/pkg/simple/client/devops"
-	esclient "kubesphere.io/kubesphere/pkg/simple/client/elasticsearch"
-	"kubesphere.io/kubesphere/pkg/simple/client/k8s"
-	"kubesphere.io/kubesphere/pkg/simple/client/mysql"
-	"kubesphere.io/kubesphere/pkg/simple/client/openpitrix"
-	"kubesphere.io/kubesphere/pkg/simple/client/prometheus"
-	"kubesphere.io/kubesphere/pkg/simple/client/s2is3"
-	"kubesphere.io/kubesphere/pkg/simple/client/servicemesh"
-	"kubesphere.io/kubesphere/pkg/simple/client/sonarqube"
+	"kubesphere.io/kubesphere/pkg/simple/client/alerting"
+	auditingclient "kubesphere.io/kubesphere/pkg/simple/client/auditing/elasticsearch"
+	"kubesphere.io/kubesphere/pkg/simple/client/cache"
+
+	"net/http"
 	"strings"
+
+	"kubesphere.io/kubesphere/pkg/simple/client/devops/jenkins"
+	eventsclient "kubesphere.io/kubesphere/pkg/simple/client/events/elasticsearch"
+	"kubesphere.io/kubesphere/pkg/simple/client/k8s"
+	esclient "kubesphere.io/kubesphere/pkg/simple/client/logging/elasticsearch"
+	"kubesphere.io/kubesphere/pkg/simple/client/monitoring/metricsserver"
+	"kubesphere.io/kubesphere/pkg/simple/client/monitoring/prometheus"
+	"kubesphere.io/kubesphere/pkg/simple/client/s3"
+	fakes3 "kubesphere.io/kubesphere/pkg/simple/client/s3/fake"
+	"kubesphere.io/kubesphere/pkg/simple/client/sonarqube"
 )
 
 type ServerRunOptions struct {
+	ConfigFile              string
 	GenericServerRunOptions *genericoptions.ServerRunOptions
+	*apiserverconfig.Config
 
-	KubernetesOptions  *k8s.KubernetesOptions
-	DevopsOptions      *devops.DevopsOptions
-	SonarQubeOptions   *sonarqube.SonarQubeOptions
-	ServiceMeshOptions *servicemesh.ServiceMeshOptions
-	MySQLOptions       *mysql.MySQLOptions
-	MonitoringOptions  *prometheus.PrometheusOptions
-	S3Options          *s2is3.S3Options
-	OpenPitrixOptions  *openpitrix.OpenPitrixOptions
-	LoggingOptions     *esclient.ElasticSearchOptions
+	//
+	DebugMode bool
 }
 
 func NewServerRunOptions() *ServerRunOptions {
-
-	s := ServerRunOptions{
+	s := &ServerRunOptions{
 		GenericServerRunOptions: genericoptions.NewServerRunOptions(),
-		KubernetesOptions:       k8s.NewKubernetesOptions(),
-		DevopsOptions:           devops.NewDevopsOptions(),
-		SonarQubeOptions:        sonarqube.NewSonarQubeOptions(),
-		ServiceMeshOptions:      servicemesh.NewServiceMeshOptions(),
-		MySQLOptions:            mysql.NewMySQLOptions(),
-		MonitoringOptions:       prometheus.NewPrometheusOptions(),
-		S3Options:               s2is3.NewS3Options(),
-		OpenPitrixOptions:       openpitrix.NewOpenPitrixOptions(),
-		LoggingOptions:          esclient.NewElasticSearchOptions(),
+		Config:                  apiserverconfig.New(),
 	}
 
-	return &s
+	return s
 }
 
 func (s *ServerRunOptions) Flags() (fss cliflag.NamedFlagSets) {
+	fs := fss.FlagSet("generic")
+	fs.BoolVar(&s.DebugMode, "debug", false, "Don't enable this if you don't know what it means.")
+	s.GenericServerRunOptions.AddFlags(fs, s.GenericServerRunOptions)
+	s.KubernetesOptions.AddFlags(fss.FlagSet("kubernetes"), s.KubernetesOptions)
+	s.AuthenticationOptions.AddFlags(fss.FlagSet("authentication"), s.AuthenticationOptions)
+	s.AuthorizationOptions.AddFlags(fss.FlagSet("authorization"), s.AuthorizationOptions)
+	s.DevopsOptions.AddFlags(fss.FlagSet("devops"), s.DevopsOptions)
+	s.SonarQubeOptions.AddFlags(fss.FlagSet("sonarqube"), s.SonarQubeOptions)
+	s.RedisOptions.AddFlags(fss.FlagSet("redis"), s.RedisOptions)
+	s.S3Options.AddFlags(fss.FlagSet("s3"), s.S3Options)
+	s.OpenPitrixOptions.AddFlags(fss.FlagSet("openpitrix"), s.OpenPitrixOptions)
+	s.NetworkOptions.AddFlags(fss.FlagSet("network"), s.NetworkOptions)
+	s.ServiceMeshOptions.AddFlags(fss.FlagSet("servicemesh"), s.ServiceMeshOptions)
+	s.MonitoringOptions.AddFlags(fss.FlagSet("monitoring"), s.MonitoringOptions)
+	s.LoggingOptions.AddFlags(fss.FlagSet("logging"), s.LoggingOptions)
+	s.MultiClusterOptions.AddFlags(fss.FlagSet("multicluster"), s.MultiClusterOptions)
+	s.EventsOptions.AddFlags(fss.FlagSet("events"), s.EventsOptions)
+	s.AuditingOptions.AddFlags(fss.FlagSet("auditing"), s.AuditingOptions)
+	s.AlertingOptions.AddFlags(fss.FlagSet("alerting"), s.AlertingOptions)
 
-	s.GenericServerRunOptions.AddFlags(fss.FlagSet("generic"))
-	s.KubernetesOptions.AddFlags(fss.FlagSet("kubernetes"))
-	s.MySQLOptions.AddFlags(fss.FlagSet("mysql"))
-	s.DevopsOptions.AddFlags(fss.FlagSet("devops"))
-	s.SonarQubeOptions.AddFlags(fss.FlagSet("sonarqube"))
-	s.S3Options.AddFlags(fss.FlagSet("s3"))
-	s.OpenPitrixOptions.AddFlags(fss.FlagSet("openpitrix"))
-	s.ServiceMeshOptions.AddFlags(fss.FlagSet("servicemesh"))
-	s.MonitoringOptions.AddFlags(fss.FlagSet("monitoring"))
-	s.LoggingOptions.AddFlags(fss.FlagSet("logging"))
-
-	fs := fss.FlagSet("klog")
+	fs = fss.FlagSet("klog")
 	local := flag.NewFlagSet("klog", flag.ExitOnError)
 	klog.InitFlags(local)
 	local.VisitAll(func(fl *flag.Flag) {
@@ -72,3 +98,137 @@ func (s *ServerRunOptions) Flags() (fss cliflag.NamedFlagSets) {
 
 	return fss
 }
+
+const fakeInterface string = "FAKE"
+
+// NewAPIServer creates an APIServer instance using given options
+func (s *ServerRunOptions) NewAPIServer(stopCh <-chan struct{}) (*apiserver.APIServer, error) {
+	apiServer := &apiserver.APIServer{
+		Config: s.Config,
+	}
+
+	kubernetesClient, err := k8s.NewKubernetesClient(s.KubernetesOptions)
+	if err != nil {
+		return nil, err
+	}
+	apiServer.KubernetesClient = kubernetesClient
+
+	informerFactory := informers.NewInformerFactories(kubernetesClient.Kubernetes(), kubernetesClient.KubeSphere(),
+		kubernetesClient.Istio(), kubernetesClient.Snapshot(), kubernetesClient.ApiExtensions(), kubernetesClient.Prometheus())
+	apiServer.InformerFactory = informerFactory
+
+	if s.MonitoringOptions == nil || len(s.MonitoringOptions.Endpoint) == 0 {
+		return nil, fmt.Errorf("moinitoring service address in configuration MUST not be empty, please check configmap/kubesphere-config in kubesphere-system namespace")
+	} else {
+		monitoringClient, err := prometheus.NewPrometheus(s.MonitoringOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connect to prometheus, please check prometheus status, error: %v", err)
+		}
+		apiServer.MonitoringClient = monitoringClient
+	}
+
+	apiServer.MetricsClient = metricsserver.NewMetricsClient(kubernetesClient.Kubernetes(), s.KubernetesOptions)
+
+	if s.LoggingOptions.Host != "" {
+		loggingClient, err := esclient.NewClient(s.LoggingOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connect to elasticsearch, please check elasticsearch status, error: %v", err)
+		}
+		apiServer.LoggingClient = loggingClient
+	}
+
+	if s.S3Options.Endpoint != "" {
+		if s.S3Options.Endpoint == fakeInterface && s.DebugMode {
+			apiServer.S3Client = fakes3.NewFakeS3()
+		} else {
+			s3Client, err := s3.NewS3Client(s.S3Options)
+			if err != nil {
+				return nil, fmt.Errorf("failed to connect to s3, please check s3 service status, error: %v", err)
+			}
+			apiServer.S3Client = s3Client
+		}
+	}
+
+	if s.DevopsOptions.Host != "" {
+		devopsClient, err := jenkins.NewDevopsClient(s.DevopsOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connect to jenkins, please check jenkins status, error: %v", err)
+		}
+		apiServer.DevopsClient = devopsClient
+	}
+
+	if s.SonarQubeOptions.Host != "" {
+		sonarClient, err := sonarqube.NewSonarQubeClient(s.SonarQubeOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connecto to sonarqube, please check sonarqube status, error: %v", err)
+		}
+		apiServer.SonarClient = sonarqube.NewSonar(sonarClient.SonarQube())
+	}
+
+	var cacheClient cache.Interface
+	if s.RedisOptions != nil && len(s.RedisOptions.Host) != 0 {
+		if s.RedisOptions.Host == fakeInterface && s.DebugMode {
+			apiServer.CacheClient = cache.NewSimpleCache()
+		} else {
+			cacheClient, err = cache.NewRedisClient(s.RedisOptions, stopCh)
+			if err != nil {
+				return nil, fmt.Errorf("failed to connect to redis service, please check redis status, error: %v", err)
+			}
+			apiServer.CacheClient = cacheClient
+		}
+	} else {
+		klog.Warning("ks-apiserver starts without redis provided, it will use in memory cache. " +
+			"This may cause inconsistencies when running ks-apiserver with multiple replicas.")
+		apiServer.CacheClient = cache.NewSimpleCache()
+	}
+
+	if s.EventsOptions.Host != "" {
+		eventsClient, err := eventsclient.NewClient(s.EventsOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connect to elasticsearch, please check elasticsearch status, error: %v", err)
+		}
+		apiServer.EventsClient = eventsClient
+	}
+
+	if s.AuditingOptions.Host != "" {
+		auditingClient, err := auditingclient.NewClient(s.AuditingOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to connect to elasticsearch, please check elasticsearch status, error: %v", err)
+		}
+		apiServer.AuditingClient = auditingClient
+	}
+
+	if s.AlertingOptions != nil && (s.AlertingOptions.PrometheusEndpoint != "" || s.AlertingOptions.ThanosRulerEndpoint != "") {
+		alertingClient, err := alerting.NewRuleClient(s.AlertingOptions)
+		if err != nil {
+			return nil, fmt.Errorf("failed to init alerting client: %v", err)
+		}
+		apiServer.AlertingClient = alertingClient
+	}
+
+	server := &http.Server{
+		Addr: fmt.Sprintf(":%d", s.GenericServerRunOptions.InsecurePort),
+	}
+
+	if s.GenericServerRunOptions.SecurePort != 0 {
+		certificate, err := tls.LoadX509KeyPair(s.GenericServerRunOptions.TlsCertFile, s.GenericServerRunOptions.TlsPrivateKey)
+		if err != nil {
+			return nil, err
+		}
+		server.TLSConfig.Certificates = []tls.Certificate{certificate}
+	}
+
+	sch := scheme.Scheme
+	if err := apis.AddToScheme(sch); err != nil {
+		klog.Fatalf("unable add APIs to scheme: %v", err)
+	}
+
+	apiServer.RuntimeCache, err = runtimecache.New(apiServer.KubernetesClient.Config(), runtimecache.Options{Scheme: sch})
+	if err != nil {
+		klog.Fatalf("unable to create runtime cache: %v", err)
+	}
+
+	apiServer.Server = server
+
+	return apiServer, nil
+}

cmd/ks-apiserver/app/options/validation.go

@@ -1,3 +1,19 @@
+/*
+Copyright 2020 KubeSphere Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
 package options
 
 // Validate validates server run options, to find
@@ -5,15 +21,21 @@ package options
 func (s *ServerRunOptions) Validate() []error {
 	var errors []error
 
+	errors = append(errors, s.GenericServerRunOptions.Validate()...)
 	errors = append(errors, s.DevopsOptions.Validate()...)
 	errors = append(errors, s.KubernetesOptions.Validate()...)
-	errors = append(errors, s.MySQLOptions.Validate()...)
 	errors = append(errors, s.ServiceMeshOptions.Validate()...)
 	errors = append(errors, s.MonitoringOptions.Validate()...)
 	errors = append(errors, s.SonarQubeOptions.Validate()...)
 	errors = append(errors, s.S3Options.Validate()...)
 	errors = append(errors, s.OpenPitrixOptions.Validate()...)
+	errors = append(errors, s.NetworkOptions.Validate()...)
 	errors = append(errors, s.LoggingOptions.Validate()...)
+	errors = append(errors, s.AuthenticationOptions.Validate()...)
+	errors = append(errors, s.AuthorizationOptions.Validate()...)
+	errors = append(errors, s.EventsOptions.Validate()...)
+	errors = append(errors, s.AuditingOptions.Validate()...)
+	errors = append(errors, s.AlertingOptions.Validate()...)
 
 	return errors
 }

cmd/ks-apiserver/app/server.go

@@ -1,75 +1,69 @@
 /*
+Copyright 2019 The KubeSphere Authors.
 
- Copyright 2019 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
 
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
-     http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
+    http://www.apache.org/licenses/LICENSE-2.0
 
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
 */
+
 package app
 
 import (
 	"fmt"
-	kconfig "github.com/kiali/kiali/config"
+
 	"github.com/spf13/cobra"
-	"k8s.io/apimachinery/pkg/runtime/schema"
 	utilerrors "k8s.io/apimachinery/pkg/util/errors"
 	cliflag "k8s.io/component-base/cli/flag"
 	"k8s.io/klog"
+
 	"kubesphere.io/kubesphere/cmd/ks-apiserver/app/options"
-	"kubesphere.io/kubesphere/pkg/apiserver/runtime"
-	"kubesphere.io/kubesphere/pkg/apiserver/servicemesh/tracing"
-	"kubesphere.io/kubesphere/pkg/informers"
-	"kubesphere.io/kubesphere/pkg/server"
-	apiserverconfig "kubesphere.io/kubesphere/pkg/server/config"
-	"kubesphere.io/kubesphere/pkg/server/filter"
-	"kubesphere.io/kubesphere/pkg/simple/client"
+	apiserverconfig "kubesphere.io/kubesphere/pkg/apiserver/config"
 	"kubesphere.io/kubesphere/pkg/utils/signals"
 	"kubesphere.io/kubesphere/pkg/utils/term"
-	"net/http"
+	"kubesphere.io/kubesphere/pkg/version"
 
-	"kubesphere.io/kubesphere/pkg/apis"
+	tracing "kubesphere.io/kubesphere/pkg/kapis/servicemesh/metrics/v1alpha2"
 )
 
 func NewAPIServerCommand() *cobra.Command {
 	s := options.NewServerRunOptions()
 
+	// Load configuration from file
+	conf, err := apiserverconfig.TryLoadFromDisk()
+	if err == nil {
+		s = &options.ServerRunOptions{
+			GenericServerRunOptions: s.GenericServerRunOptions,
+			Config:                  conf,
+		}
+	} else {
+		klog.Fatal("Failed to load configuration from disk", err)
+	}
+
 	cmd := &cobra.Command{
 		Use: "ks-apiserver",
-		Long: `The KubeSphere API server validates and configures data for the api objects. 
+		Long: `The KubeSphere API server validates and configures data for the API objects. 
 The API Server services REST operations and provides the frontend to the
 cluster's shared state through which all other components interact.`,
 		RunE: func(cmd *cobra.Command, args []string) error {
-			err := apiserverconfig.Load()
-			if err != nil {
-				return err
-			}
-
-			err = Complete(s)
-			if err != nil {
-				return err
-			}
-
 			if errs := s.Validate(); len(errs) != 0 {
 				return utilerrors.NewAggregate(errs)
 			}
 
 			return Run(s, signals.SetupSignalHandler())
 		},
+		SilenceUsage: true,
 	}
 
 	fs := cmd.Flags()
 	namedFlagSets := s.Flags()
-
 	for _, f := range namedFlagSets.FlagSets {
 		fs.AddFlagSet(f)
 	}
@@ -80,274 +74,45 @@ cluster's shared state through which all other components interact.`,
 		fmt.Fprintf(cmd.OutOrStdout(), "%s\n\n"+usageFmt, cmd.Long, cmd.UseLine())
 		cliflag.PrintSections(cmd.OutOrStdout(), namedFlagSets, cols)
 	})
+
+	versionCmd := &cobra.Command{
+		Use:   "version",
+		Short: "Print the version of KubeSphere ks-apiserver",
+		Run: func(cmd *cobra.Command, args []string) {
+			cmd.Println(version.Get())
+		},
+	}
+
+	cmd.AddCommand(versionCmd)
+
 	return cmd
 }
 
 func Run(s *options.ServerRunOptions, stopCh <-chan struct{}) error {
 
-	err := CreateClientSet(apiserverconfig.Get(), stopCh)
-	if err != nil {
-		return err
-	}
-
-	err = WaitForResourceSync(stopCh)
-	if err != nil {
-		return err
-	}
-
 	initializeServicemeshConfig(s)
 
-	err = CreateAPIServer(s)
+	apiserver, err := s.NewAPIServer(stopCh)
 	if err != nil {
 		return err
 	}
 
-	return nil
+	err = apiserver.PrepareRun(stopCh)
+	if err != nil {
+		return nil
+	}
+
+	return apiserver.Run(stopCh)
 }
 
 func initializeServicemeshConfig(s *options.ServerRunOptions) {
-	// Initialize kiali config
-	config := kconfig.NewConfig()
+	// Config jaeger query endpoint address
+	if s.ServiceMeshOptions != nil && len(s.ServiceMeshOptions.JaegerQueryHost) != 0 {
+		tracing.JaegerQueryUrl = s.ServiceMeshOptions.JaegerQueryHost
+	}
 
-	tracing.JaegerQueryUrl = s.ServiceMeshOptions.JaegerQueryHost
-
-	// Exclude system namespaces
-	config.API.Namespaces.Exclude = []string{"istio-system", "kubesphere*", "kube*"}
-	config.InCluster = true
-
-	// Set default prometheus service url
-	config.ExternalServices.PrometheusServiceURL = s.ServiceMeshOptions.ServicemeshPrometheusHost
-	config.ExternalServices.PrometheusCustomMetricsURL = config.ExternalServices.PrometheusServiceURL
-
-	// Set istio pilot discovery service url
-	config.ExternalServices.Istio.UrlServiceVersion = s.ServiceMeshOptions.IstioPilotHost
-
-	kconfig.Set(config)
-}
-
-//
-func CreateAPIServer(s *options.ServerRunOptions) error {
-	var err error
-
-	container := runtime.Container
-	container.DoNotRecover(false)
-	container.Filter(filter.Logging)
-	container.RecoverHandler(server.LogStackOnRecover)
-
-	apis.InstallAPIs(container)
-
-	// install config api
-	apiserverconfig.InstallAPI(container)
-
-	if s.GenericServerRunOptions.InsecurePort != 0 {
-		err = http.ListenAndServe(fmt.Sprintf("%s:%d", s.GenericServerRunOptions.BindAddress, s.GenericServerRunOptions.InsecurePort), container)
-		if err == nil {
-			klog.V(0).Infof("Server listening on insecure port %d.", s.GenericServerRunOptions.InsecurePort)
-		}
-	}
-
-	if s.GenericServerRunOptions.SecurePort != 0 && len(s.GenericServerRunOptions.TlsCertFile) > 0 && len(s.GenericServerRunOptions.TlsPrivateKey) > 0 {
-		err = http.ListenAndServeTLS(fmt.Sprintf("%s:%d", s.GenericServerRunOptions.BindAddress, s.GenericServerRunOptions.SecurePort), s.GenericServerRunOptions.TlsCertFile, s.GenericServerRunOptions.TlsPrivateKey, container)
-		if err == nil {
-			klog.V(0).Infof("Server listening on secure port %d.", s.GenericServerRunOptions.SecurePort)
-		}
-	}
-
-	return err
-}
-
-func CreateClientSet(conf *apiserverconfig.Config, stopCh <-chan struct{}) error {
-	csop := &client.ClientSetOptions{}
-
-	csop.SetDevopsOptions(conf.DevopsOptions).
-		SetSonarQubeOptions(conf.SonarQubeOptions).
-		SetKubernetesOptions(conf.KubernetesOptions).
-		SetMySQLOptions(conf.MySQLOptions).
-		SetLdapOptions(conf.LdapOptions).
-		SetS3Options(conf.S3Options).
-		SetOpenPitrixOptions(conf.OpenPitrixOptions).
-		SetPrometheusOptions(conf.MonitoringOptions).
-		SetKubeSphereOptions(conf.KubeSphereOptions).
-		SetElasticSearchOptions(conf.LoggingOptions)
-
-	client.NewClientSetFactory(csop, stopCh)
-
-	return nil
-}
-
-func WaitForResourceSync(stopCh <-chan struct{}) error {
-	klog.V(0).Info("Start cache objects")
-
-	discoveryClient := client.ClientSets().K8s().Discovery()
-	apiResourcesList, err := discoveryClient.ServerResources()
-	if err != nil {
-		return err
-	}
-
-	isResourceExists := func(resource schema.GroupVersionResource) bool {
-		for _, apiResource := range apiResourcesList {
-			if apiResource.GroupVersion == resource.GroupVersion().String() {
-				for _, rsc := range apiResource.APIResources {
-					if rsc.Name == resource.Resource {
-						return true
-					}
-				}
-			}
-		}
-		return false
-	}
-
-	informerFactory := informers.SharedInformerFactory()
-
-	// resources we have to create informer first
-	k8sGVRs := []schema.GroupVersionResource{
-		{Group: "", Version: "v1", Resource: "namespaces"},
-		{Group: "", Version: "v1", Resource: "nodes"},
-		{Group: "", Version: "v1", Resource: "resourcequotas"},
-		{Group: "", Version: "v1", Resource: "pods"},
-		{Group: "", Version: "v1", Resource: "services"},
-		{Group: "", Version: "v1", Resource: "persistentvolumeclaims"},
-		{Group: "", Version: "v1", Resource: "secrets"},
-		{Group: "", Version: "v1", Resource: "configmaps"},
-
-		{Group: "rbac.authorization.k8s.io", Version: "v1", Resource: "roles"},
-		{Group: "rbac.authorization.k8s.io", Version: "v1", Resource: "rolebindings"},
-		{Group: "rbac.authorization.k8s.io", Version: "v1", Resource: "clusterroles"},
-		{Group: "rbac.authorization.k8s.io", Version: "v1", Resource: "clusterrolebindings"},
-
-		{Group: "apps", Version: "v1", Resource: "deployments"},
-		{Group: "apps", Version: "v1", Resource: "daemonsets"},
-		{Group: "apps", Version: "v1", Resource: "replicasets"},
-		{Group: "apps", Version: "v1", Resource: "statefulsets"},
-		{Group: "apps", Version: "v1", Resource: "controllerrevisions"},
-
-		{Group: "storage.k8s.io", Version: "v1", Resource: "storageclasses"},
-
-		{Group: "batch", Version: "v1", Resource: "jobs"},
-		{Group: "batch", Version: "v1beta1", Resource: "cronjobs"},
-
-		{Group: "extensions", Version: "v1beta1", Resource: "ingresses"},
-
-		{Group: "autoscaling", Version: "v2beta2", Resource: "horizontalpodautoscalers"},
-	}
-
-	for _, gvr := range k8sGVRs {
-		if !isResourceExists(gvr) {
-			klog.Warningf("resource %s not exists in the cluster", gvr)
-		} else {
-			_, err := informerFactory.ForResource(gvr)
-			if err != nil {
-				klog.Errorf("cannot create informer for %s", gvr)
-				return err
-			}
-		}
-	}
-
-	informerFactory.Start(stopCh)
-	informerFactory.WaitForCacheSync(stopCh)
-
-	s2iInformerFactory := informers.S2iSharedInformerFactory()
-
-	s2iGVRs := []schema.GroupVersionResource{
-		{Group: "devops.kubesphere.io", Version: "v1alpha1", Resource: "s2ibuildertemplates"},
-		{Group: "devops.kubesphere.io", Version: "v1alpha1", Resource: "s2iruns"},
-		{Group: "devops.kubesphere.io", Version: "v1alpha1", Resource: "s2ibuilders"},
-	}
-
-	for _, gvr := range s2iGVRs {
-		if !isResourceExists(gvr) {
-			klog.Warningf("resource %s not exists in the cluster", gvr)
-		} else {
-			_, err := s2iInformerFactory.ForResource(gvr)
-			if err != nil {
-				return err
-			}
-		}
-	}
-
-	s2iInformerFactory.Start(stopCh)
-	s2iInformerFactory.WaitForCacheSync(stopCh)
-
-	ksInformerFactory := informers.KsSharedInformerFactory()
-
-	ksGVRs := []schema.GroupVersionResource{
-		{Group: "tenant.kubesphere.io", Version: "v1alpha1", Resource: "workspaces"},
-		{Group: "devops.kubesphere.io", Version: "v1alpha1", Resource: "s2ibinaries"},
-
-		{Group: "servicemesh.kubesphere.io", Version: "v1alpha2", Resource: "strategies"},
-		{Group: "servicemesh.kubesphere.io", Version: "v1alpha2", Resource: "servicepolicies"},
-	}
-
-	for _, gvr := range ksGVRs {
-		if !isResourceExists(gvr) {
-			klog.Warningf("resource %s not exists in the cluster", gvr)
-		} else {
-			_, err := ksInformerFactory.ForResource(gvr)
-			if err != nil {
-				return err
-			}
-		}
-	}
-
-	ksInformerFactory.Start(stopCh)
-	ksInformerFactory.WaitForCacheSync(stopCh)
-
-	appInformerFactory := informers.AppSharedInformerFactory()
-
-	appGVRs := []schema.GroupVersionResource{
-		{Group: "app.k8s.io", Version: "v1beta1", Resource: "applications"},
-	}
-
-	for _, gvr := range appGVRs {
-		if !isResourceExists(gvr) {
-			klog.Warningf("resource %s not exists in the cluster", gvr)
-		} else {
-			_, err := appInformerFactory.ForResource(gvr)
-			if err != nil {
-				return err
-			}
-		}
-	}
-
-	appInformerFactory.Start(stopCh)
-	appInformerFactory.WaitForCacheSync(stopCh)
-
-	klog.V(0).Info("Finished caching objects")
-
-	return nil
-
-}
-
-// apply server run options to configuration
-func Complete(s *options.ServerRunOptions) error {
-
-	// loading configuration file
-	conf := apiserverconfig.Get()
-
-	conf.Apply(&apiserverconfig.Config{
-		MySQLOptions:       s.MySQLOptions,
-		DevopsOptions:      s.DevopsOptions,
-		SonarQubeOptions:   s.SonarQubeOptions,
-		KubernetesOptions:  s.KubernetesOptions,
-		ServiceMeshOptions: s.ServiceMeshOptions,
-		MonitoringOptions:  s.MonitoringOptions,
-		S3Options:          s.S3Options,
-		OpenPitrixOptions:  s.OpenPitrixOptions,
-		LoggingOptions:     s.LoggingOptions,
-	})
-
-	*s = options.ServerRunOptions{
-		GenericServerRunOptions: s.GenericServerRunOptions,
-		KubernetesOptions:       conf.KubernetesOptions,
-		DevopsOptions:           conf.DevopsOptions,
-		SonarQubeOptions:        conf.SonarQubeOptions,
-		ServiceMeshOptions:      conf.ServiceMeshOptions,
-		MySQLOptions:            conf.MySQLOptions,
-		MonitoringOptions:       conf.MonitoringOptions,
-		S3Options:               conf.S3Options,
-		OpenPitrixOptions:       conf.OpenPitrixOptions,
-		LoggingOptions:          conf.LoggingOptions,
-	}
-
-	return nil
+	// Set the kiali query endpoint address
+	if s.ServiceMeshOptions != nil && len(s.ServiceMeshOptions.KialiQueryHost) != 0 {
+		tracing.KialiQueryUrl = s.ServiceMeshOptions.KialiQueryHost
+	}
 }

cmd/ks-iam/apiserver.go

@@ -1,32 +0,0 @@
-/*
-
- Copyright 2019 The KubeSphere Authors.
-
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
-     http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
-*/
-package main
-
-import (
-	"kubesphere.io/kubesphere/cmd/ks-iam/app"
-	"log"
-)
-
-func main() {
-
-	cmd := app.NewAPIServerCommand()
-
-	if err := cmd.Execute(); err != nil {
-		log.Fatalln(err)
-	}
-}

cmd/ks-iam/app/options/options.go

@@ -1,86 +0,0 @@
-/*
-
- Copyright 2019 The KubeSphere Authors.
-
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
-     http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
-*/
-package options
-
-import (
-	"flag"
-	cliflag "k8s.io/component-base/cli/flag"
-	"k8s.io/klog"
-	genericoptions "kubesphere.io/kubesphere/pkg/server/options"
-	"kubesphere.io/kubesphere/pkg/simple/client/k8s"
-	"kubesphere.io/kubesphere/pkg/simple/client/ldap"
-	"kubesphere.io/kubesphere/pkg/simple/client/mysql"
-	"kubesphere.io/kubesphere/pkg/simple/client/redis"
-	"strings"
-	"time"
-)
-
-type ServerRunOptions struct {
-	GenericServerRunOptions *genericoptions.ServerRunOptions
-	KubernetesOptions       *k8s.KubernetesOptions
-	LdapOptions             *ldap.LdapOptions
-	RedisOptions            *redis.RedisOptions
-	MySQLOptions            *mysql.MySQLOptions
-	AdminEmail              string
-	AdminPassword           string
-	TokenIdleTimeout        time.Duration
-	JWTSecret               string
-	AuthRateLimit           string
-	EnableMultiLogin        bool
-	GenerateKubeConfig      bool
-}
-
-func NewServerRunOptions() *ServerRunOptions {
-	s := &ServerRunOptions{
-		GenericServerRunOptions: genericoptions.NewServerRunOptions(),
-		KubernetesOptions:       k8s.NewKubernetesOptions(),
-		LdapOptions:             ldap.NewLdapOptions(),
-		MySQLOptions:            mysql.NewMySQLOptions(),
-		RedisOptions:            redis.NewRedisOptions(),
-	}
-	return s
-}
-
-func (s *ServerRunOptions) Flags() (fss cliflag.NamedFlagSets) {
-
-	fs := fss.FlagSet("generic")
-
-	s.GenericServerRunOptions.AddFlags(fs)
-	fs.StringVar(&s.AdminEmail, "admin-email", "admin@kubesphere.io", "default administrator's email")
-	fs.StringVar(&s.AdminPassword, "admin-password", "passw0rd", "default administrator's password")
-	fs.DurationVar(&s.TokenIdleTimeout, "token-idle-timeout", 30*time.Minute, "tokens that are idle beyond that time will expire,0s means the token has no expiration time. valid time units are \"ns\",\"us\",\"ms\",\"s\",\"m\",\"h\"")
-	fs.StringVar(&s.JWTSecret, "jwt-secret", "", "jwt secret")
-	fs.StringVar(&s.AuthRateLimit, "auth-rate-limit", "5/30m", "specifies the maximum number of authentication attempts permitted and time interval,valid time units are \"s\",\"m\",\"h\"")
-	fs.BoolVar(&s.EnableMultiLogin, "enable-multi-login", false, "allow one account to have multiple sessions")
-	fs.BoolVar(&s.GenerateKubeConfig, "generate-kubeconfig", true, "generate kubeconfig for new users, kubeconfig is required in devops pipeline, set to false if you don't need devops.")
-
-	s.KubernetesOptions.AddFlags(fss.FlagSet("kubernetes"))
-	s.LdapOptions.AddFlags(fss.FlagSet("ldap"))
-	s.RedisOptions.AddFlags(fss.FlagSet("redis"))
-	s.MySQLOptions.AddFlags(fss.FlagSet("mysql"))
-
-	kfs := fss.FlagSet("klog")
-	local := flag.NewFlagSet("klog", flag.ExitOnError)
-	klog.InitFlags(local)
-	local.VisitAll(func(fl *flag.Flag) {
-		fl.Name = strings.Replace(fl.Name, "_", "-", -1)
-		kfs.AddGoFlag(fl)
-	})
-
-	return fss
-}

cmd/ks-iam/app/options/validation.go

@@ -1,11 +0,0 @@
-package options
-
-func (s *ServerRunOptions) Validate() []error {
-	errs := []error{}
-
-	errs = append(errs, s.KubernetesOptions.Validate()...)
-	errs = append(errs, s.GenericServerRunOptions.Validate()...)
-	errs = append(errs, s.LdapOptions.Validate()...)
-
-	return errs
-}

cmd/ks-iam/app/server.go

@@ -1,161 +0,0 @@
-/*
-
- Copyright 2019 The KubeSphere Authors.
-
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
-     http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
-*/
-package app
-
-import (
-	"fmt"
-	"github.com/spf13/cobra"
-	utilerrors "k8s.io/apimachinery/pkg/util/errors"
-	cliflag "k8s.io/component-base/cli/flag"
-	"k8s.io/klog"
-	"kubesphere.io/kubesphere/cmd/ks-iam/app/options"
-	"kubesphere.io/kubesphere/pkg/apis"
-	"kubesphere.io/kubesphere/pkg/apiserver/runtime"
-	"kubesphere.io/kubesphere/pkg/informers"
-	"kubesphere.io/kubesphere/pkg/models/iam"
-	"kubesphere.io/kubesphere/pkg/server"
-	apiserverconfig "kubesphere.io/kubesphere/pkg/server/config"
-	"kubesphere.io/kubesphere/pkg/server/filter"
-	"kubesphere.io/kubesphere/pkg/simple/client"
-	"kubesphere.io/kubesphere/pkg/utils/jwtutil"
-	"kubesphere.io/kubesphere/pkg/utils/signals"
-	"kubesphere.io/kubesphere/pkg/utils/term"
-	"net/http"
-)
-
-func NewAPIServerCommand() *cobra.Command {
-	s := options.NewServerRunOptions()
-
-	cmd := &cobra.Command{
-		Use: "ks-iam",
-		Long: `The KubeSphere account server validates and configures data
-for the api objects. The API Server services REST operations and provides the frontend to the
-cluster's shared state through which all other components interact.`,
-		RunE: func(cmd *cobra.Command, args []string) error {
-
-			err := apiserverconfig.Load()
-			if err != nil {
-				return err
-			}
-
-			err = Complete(s)
-			if err != nil {
-				return err
-			}
-
-			if errs := s.Validate(); len(errs) != 0 {
-				return utilerrors.NewAggregate(errs)
-			}
-
-			return Run(s, signals.SetupSignalHandler())
-		},
-	}
-
-	fs := cmd.Flags()
-	namedFlagSets := s.Flags()
-
-	for _, f := range namedFlagSets.FlagSets {
-		fs.AddFlagSet(f)
-	}
-
-	usageFmt := "Usage:\n  %s\n"
-	cols, _, _ := term.TerminalSize(cmd.OutOrStdout())
-	cmd.SetHelpFunc(func(cmd *cobra.Command, args []string) {
-		fmt.Fprintf(cmd.OutOrStdout(), "%s\n\n"+usageFmt, cmd.Long, cmd.UseLine())
-		cliflag.PrintSections(cmd.OutOrStdout(), namedFlagSets, cols)
-	})
-
-	return cmd
-}
-
-func Run(s *options.ServerRunOptions, stopChan <-chan struct{}) error {
-	csop := client.NewClientSetOptions()
-	csop.SetKubernetesOptions(s.KubernetesOptions).
-		SetLdapOptions(s.LdapOptions).
-		SetRedisOptions(s.RedisOptions).
-		SetMySQLOptions(s.MySQLOptions)
-
-	client.NewClientSetFactory(csop, stopChan)
-
-	waitForResourceSync(stopChan)
-
-	err := iam.Init(s.AdminEmail, s.AdminPassword, s.AuthRateLimit, s.TokenIdleTimeout, s.EnableMultiLogin, s.GenerateKubeConfig)
-
-	jwtutil.Setup(s.JWTSecret)
-
-	if err != nil {
-		return err
-	}
-
-	container := runtime.Container
-	container.Filter(filter.Logging)
-	container.DoNotRecover(false)
-	container.RecoverHandler(server.LogStackOnRecover)
-
-	apis.InstallAuthorizationAPIs(container)
-
-	if s.GenericServerRunOptions.InsecurePort != 0 {
-		klog.Infof("Server listening on %s:%d ", s.GenericServerRunOptions.BindAddress, s.GenericServerRunOptions.InsecurePort)
-		err = http.ListenAndServe(fmt.Sprintf("%s:%d", s.GenericServerRunOptions.BindAddress, s.GenericServerRunOptions.InsecurePort), container)
-	}
-
-	if s.GenericServerRunOptions.SecurePort != 0 && len(s.GenericServerRunOptions.TlsCertFile) > 0 && len(s.GenericServerRunOptions.TlsPrivateKey) > 0 {
-		klog.Infof("Server listening on %s:%d", s.GenericServerRunOptions.BindAddress, s.GenericServerRunOptions.SecurePort)
-		err = http.ListenAndServeTLS(fmt.Sprintf("%s:%d", s.GenericServerRunOptions.BindAddress, s.GenericServerRunOptions.SecurePort), s.GenericServerRunOptions.TlsCertFile, s.GenericServerRunOptions.TlsPrivateKey, container)
-	}
-
-	return err
-}
-
-func Complete(s *options.ServerRunOptions) error {
-	conf := apiserverconfig.Get()
-
-	conf.Apply(&apiserverconfig.Config{
-		KubernetesOptions: s.KubernetesOptions,
-		LdapOptions:       s.LdapOptions,
-		RedisOptions:      s.RedisOptions,
-		MySQLOptions:      s.MySQLOptions,
-	})
-
-	s.KubernetesOptions = conf.KubernetesOptions
-	s.LdapOptions = conf.LdapOptions
-	s.RedisOptions = conf.RedisOptions
-	s.MySQLOptions = conf.MySQLOptions
-
-	return nil
-}
-
-func waitForResourceSync(stopCh <-chan struct{}) {
-
-	informerFactory := informers.SharedInformerFactory()
-	informerFactory.Rbac().V1().Roles().Lister()
-	informerFactory.Rbac().V1().RoleBindings().Lister()
-	informerFactory.Rbac().V1().ClusterRoles().Lister()
-	informerFactory.Rbac().V1().ClusterRoleBindings().Lister()
-
-	informerFactory.Core().V1().Namespaces().Lister()
-
-	informerFactory.Start(stopCh)
-	informerFactory.WaitForCacheSync(stopCh)
-
-	ksInformerFactory := informers.KsSharedInformerFactory()
-	ksInformerFactory.Tenant().V1alpha1().Workspaces().Lister()
-
-	ksInformerFactory.Start(stopCh)
-	ksInformerFactory.WaitForCacheSync(stopCh)
-}

cmd/ks-network/main.go

@@ -1,25 +0,0 @@
-package main
-
-import (
-	"flag"
-
-	"k8s.io/klog"
-	"kubesphere.io/kubesphere/pkg/controller/network/runoption"
-)
-
-var opt runoption.RunOption
-
-func init() {
-	flag.StringVar(&opt.ProviderName, "np-provider", "calico", "specify the network policy provider, k8s or calico")
-	flag.BoolVar(&opt.AllowInsecureEtcd, "allow-insecure-etcd", false, "specify allow connect to etcd using insecure http")
-	flag.StringVar(&opt.DataStoreType, "datastore-type", "k8s", "specify the datastore type of calico")
-	//TODO add more flags
-}
-
-func main() {
-	klog.InitFlags(nil)
-	flag.Set("logtostderr", "true")
-	flag.Parse()
-	klog.V(1).Info("Preparing kubernetes client")
-	klog.Fatal(opt.Run())
-}

config/crds/app_v1beta1_application.yaml

@@ -1,236 +1,529 @@
-apiVersion: apiextensions.k8s.io/v1beta1
+# Copyright 2020 The Kubernetes Authors.
+# SPDX-License-Identifier: Apache-2.0
+
+apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
+  annotations:
+    api-approved.kubernetes.io: https://github.com/kubernetes-sigs/application/pull/2
+    controller-gen.kubebuilder.io/version: v0.4.0
   creationTimestamp: null
-  labels:
-    controller-tools.k8s.io: "1.0"
   name: applications.app.k8s.io
 spec:
   group: app.k8s.io
   names:
+    categories:
+    - all
     kind: Application
+    listKind: ApplicationList
     plural: applications
+    shortNames:
+    - app
+    singular: application
   scope: Namespaced
-  validation:
-    openAPIV3Schema:
-      properties:
-        apiVersion:
-          type: string
-        kind:
-          type: string
-        metadata:
-          type: object
-        spec:
-          properties:
-            assemblyPhase:
-              type: string
-            componentKinds:
-              items:
-                type: object
-              type: array
-            descriptor:
-              properties:
-                description:
-                  type: string
-                icons:
-                  items:
-                    properties:
-                      size:
-                        type: string
-                      src:
-                        type: string
-                      type:
-                        type: string
-                    required:
-                    - src
-                    type: object
-                  type: array
-                keywords:
-                  items:
-                    type: string
-                  type: array
-                links:
-                  items:
-                    properties:
-                      description:
-                        type: string
-                      url:
-                        type: string
-                    type: object
-                  type: array
-                maintainers:
-                  items:
-                    properties:
-                      email:
-                        type: string
-                      name:
-                        type: string
-                      url:
-                        type: string
-                    type: object
-                  type: array
-                notes:
-                  type: string
-                owners:
-                  items:
-                    properties:
-                      email:
-                        type: string
-                      name:
-                        type: string
-                      url:
-                        type: string
-                    type: object
-                  type: array
-                type:
-                  type: string
-                version:
-                  type: string
-              type: object
-            info:
-              items:
+  versions:
+  - additionalPrinterColumns:
+    - description: The type of the application
+      jsonPath: .spec.descriptor.type
+      name: Type
+      type: string
+    - description: The creation date
+      jsonPath: .spec.descriptor.version
+      name: Version
+      type: string
+    - description: The application object owns the matched resources
+      jsonPath: .spec.addOwnerRef
+      name: Owner
+      type: boolean
+    - description: Numbers of components ready
+      jsonPath: .status.componentsReady
+      name: Ready
+      type: string
+    - description: The creation date
+      jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1beta1
+    schema:
+      openAPIV3Schema:
+        description: Application is the Schema for the applications API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: ApplicationSpec defines the specification for an Application.
+            properties:
+              addOwnerRef:
+                description: AddOwnerRef objects - flag to indicate if we need to
+                  add OwnerRefs to matching objects Matching is done by using Selector
+                  to query all ComponentGroupKinds
+                type: boolean
+              assemblyPhase:
+                description: AssemblyPhase represents the current phase of the application's
+                  assembly. An empty value is equivalent to "Succeeded".
+                type: string
+              componentKinds:
+                description: ComponentGroupKinds is a list of Kinds for Application's
+                  components (e.g. Deployments, Pods, Services, CRDs). It can be used
+                  in conjunction with the Application's Selector to list or watch
+                  the Applications components.
+                items:
+                  description: GroupKind specifies a Group and a Kind, but does not
+                    force a version.  This is useful for identifying concepts during
+                    lookup stages without having partially valid types
+                  properties:
+                    group:
+                      type: string
+                    kind:
+                      type: string
+                  required:
+                  - group
+                  - kind
+                  type: object
+                type: array
+              descriptor:
+                description: Descriptor regroups information and metadata about an
+                  application.
                 properties:
-                  name:
+                  description:
+                    description: Description is a brief string description of the
+                      Application.
                     type: string
+                  icons:
+                    description: Icons is an optional list of icons for an application.
+                      Icon information includes the source, size, and mime type.
+                    items:
+                      description: ImageSpec contains information about an image used
+                        as an icon.
+                      properties:
+                        size:
+                          description: (optional) The size of the image in pixels
+                            (e.g., 25x25).
+                          type: string
+                        src:
+                          description: The source for image represented as either
+                            an absolute URL to the image or a Data URL containing
+                            the image. Data URLs are defined in RFC 2397.
+                          type: string
+                        type:
+                          description: (optional) The mine type of the image (e.g.,
+                            "image/png").
+                          type: string
+                      required:
+                      - src
+                      type: object
+                    type: array
+                  keywords:
+                    description: Keywords is an optional list of key words associated
+                      with the application (e.g. MySQL, RDBMS, database).
+                    items:
+                      type: string
+                    type: array
+                  links:
+                    description: Links are a list of descriptive URLs intended to
+                      be used to surface additional documentation, dashboards, etc.
+                    items:
+                      description: Link contains information about an URL to surface
+                        documentation, dashboards, etc.
+                      properties:
+                        description:
+                          description: Description is human readable content explaining
+                            the purpose of the link.
+                          type: string
+                        url:
+                          description: Url typically points at a website address.
+                          type: string
+                      type: object
+                    type: array
+                  maintainers:
+                    description: Maintainers is an optional list of maintainers of
+                      the application. The maintainers in this list maintain the the
+                      source code, images, and package for the application.
+                    items:
+                      description: ContactData contains information about an individual
+                        or organization.
+                      properties:
+                        email:
+                          description: Email is the email address.
+                          type: string
+                        name:
+                          description: Name is the descriptive name.
+                          type: string
+                        url:
+                          description: Url could typically be a website address.
+                          type: string
+                      type: object
+                    type: array
+                  notes:
+                    description: Notes contain a human readable snippets intended
+                      as a quick start for the users of the Application. CommonMark
+                      markdown syntax may be used for rich text representation.
+                    type: string
+                  owners:
+                    description: Owners is an optional list of the owners of the installed
+                      application. The owners of the application should be contacted
+                      in the event of a planned or unplanned disruption affecting
+                      the application.
+                    items:
+                      description: ContactData contains information about an individual
+                        or organization.
+                      properties:
+                        email:
+                          description: Email is the email address.
+                          type: string
+                        name:
+                          description: Name is the descriptive name.
+                          type: string
+                        url:
+                          description: Url could typically be a website address.
+                          type: string
+                      type: object
+                    type: array
                   type:
+                    description: Type is the type of the application (e.g. WordPress,
+                      MySQL, Cassandra).
                     type: string
-                  value:
+                  version:
+                    description: Version is an optional version indicator for the
+                      Application.
                     type: string
-                  valueFrom:
-                    properties:
-                      configMapKeyRef:
-                        properties:
-                          apiVersion:
+                type: object
+              info:
+                description: Info contains human readable key,value pairs for the
+                  Application.
+                items:
+                  description: InfoItem is a human readable key,value pair containing
+                    important information about how to access the Application.
+                  properties:
+                    name:
+                      description: Name is a human readable title for this piece of
+                        information.
+                      type: string
+                    type:
+                      description: Type of the value for this InfoItem.
+                      type: string
+                    value:
+                      description: Value is human readable content.
+                      type: string
+                    valueFrom:
+                      description: ValueFrom defines a reference to derive the value
+                        from another source.
+                      properties:
+                        configMapKeyRef:
+                          description: Selects a key of a ConfigMap.
+                          properties:
+                            apiVersion:
+                              description: API version of the referent.
+                              type: string
+                            fieldPath:
+                              description: 'If referring to a piece of an object instead
+                                of an entire object, this string should contain a
+                                valid JSON/Go field access statement, such as desiredState.manifest.containers[2].
+                                For example, if the object reference is to a container
+                                within a pod, this would take on a value like: "spec.containers{name}"
+                                (where "name" refers to the name of the container
+                                that triggered the event) or if no container name
+                                is specified "spec.containers[2]" (container with
+                                index 2 in this pod). This syntax is chosen only to
+                                have some well-defined way of referencing a part of
+                                an object. TODO: this design is not final and this
+                                field is subject to change in the future.'
+                              type: string
+                            key:
+                              description: The key to select.
+                              type: string
+                            kind:
+                              description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                              type: string
+                            name:
+                              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                              type: string
+                            namespace:
+                              description: 'Namespace of the referent. More info:
+                                https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                              type: string
+                            resourceVersion:
+                              description: 'Specific resourceVersion to which this
+                                reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                              type: string
+                            uid:
+                              description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                              type: string
+                          type: object
+                        ingressRef:
+                          description: Select an Ingress.
+                          properties:
+                            apiVersion:
+                              description: API version of the referent.
+                              type: string
+                            fieldPath:
+                              description: 'If referring to a piece of an object instead
+                                of an entire object, this string should contain a
+                                valid JSON/Go field access statement, such as desiredState.manifest.containers[2].
+                                For example, if the object reference is to a container
+                                within a pod, this would take on a value like: "spec.containers{name}"
+                                (where "name" refers to the name of the container
+                                that triggered the event) or if no container name
+                                is specified "spec.containers[2]" (container with
+                                index 2 in this pod). This syntax is chosen only to
+                                have some well-defined way of referencing a part of
+                                an object. TODO: this design is not final and this
+                                field is subject to change in the future.'
+                              type: string
+                            host:
+                              description: The optional host to select.
+                              type: string
+                            kind:
+                              description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                              type: string
+                            name:
+                              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                              type: string
+                            namespace:
+                              description: 'Namespace of the referent. More info:
+                                https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                              type: string
+                            path:
+                              description: The optional HTTP path.
+                              type: string
+                            protocol:
+                              description: Protocol for the ingress
+                              type: string
+                            resourceVersion:
+                              description: 'Specific resourceVersion to which this
+                                reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                              type: string
+                            uid:
+                              description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                              type: string
+                          type: object
+                        secretKeyRef:
+                          description: Selects a key of a Secret.
+                          properties:
+                            apiVersion:
+                              description: API version of the referent.
+                              type: string
+                            fieldPath:
+                              description: 'If referring to a piece of an object instead
+                                of an entire object, this string should contain a
+                                valid JSON/Go field access statement, such as desiredState.manifest.containers[2].
+                                For example, if the object reference is to a container
+                                within a pod, this would take on a value like: "spec.containers{name}"
+                                (where "name" refers to the name of the container
+                                that triggered the event) or if no container name
+                                is specified "spec.containers[2]" (container with
+                                index 2 in this pod). This syntax is chosen only to
+                                have some well-defined way of referencing a part of
+                                an object. TODO: this design is not final and this
+                                field is subject to change in the future.'
+                              type: string
+                            key:
+                              description: The key to select.
+                              type: string
+                            kind:
+                              description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                              type: string
+                            name:
+                              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                              type: string
+                            namespace:
+                              description: 'Namespace of the referent. More info:
+                                https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                              type: string
+                            resourceVersion:
+                              description: 'Specific resourceVersion to which this
+                                reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                              type: string
+                            uid:
+                              description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                              type: string
+                          type: object
+                        serviceRef:
+                          description: Select a Service.
+                          properties:
+                            apiVersion:
+                              description: API version of the referent.
+                              type: string
+                            fieldPath:
+                              description: 'If referring to a piece of an object instead
+                                of an entire object, this string should contain a
+                                valid JSON/Go field access statement, such as desiredState.manifest.containers[2].
+                                For example, if the object reference is to a container
+                                within a pod, this would take on a value like: "spec.containers{name}"
+                                (where "name" refers to the name of the container
+                                that triggered the event) or if no container name
+                                is specified "spec.containers[2]" (container with
+                                index 2 in this pod). This syntax is chosen only to
+                                have some well-defined way of referencing a part of
+                                an object. TODO: this design is not final and this
+                                field is subject to change in the future.'
+                              type: string
+                            kind:
+                              description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                              type: string
+                            name:
+                              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                              type: string
+                            namespace:
+                              description: 'Namespace of the referent. More info:
+                                https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                              type: string
+                            path:
+                              description: The optional HTTP path.
+                              type: string
+                            port:
+                              description: The optional port to select.
+                              format: int32
+                              type: integer
+                            protocol:
+                              description: Protocol for the service
+                              type: string
+                            resourceVersion:
+                              description: 'Specific resourceVersion to which this
+                                reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                              type: string
+                            uid:
+                              description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                              type: string
+                          type: object
+                        type:
+                          description: Type of source.
+                          type: string
+                      type: object
+                  type: object
+                type: array
+              selector:
+                description: 'Selector is a label query over kinds that created by
+                  the application. It must match the component objects'' labels. More
+                  info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors'
+                properties:
+                  matchExpressions:
+                    description: matchExpressions is a list of label selector requirements.
+                      The requirements are ANDed.
+                    items:
+                      description: A label selector requirement is a selector that
+                        contains values, a key, and an operator that relates the key
+                        and values.
+                      properties:
+                        key:
+                          description: key is the label key that the selector applies
+                            to.
+                          type: string
+                        operator:
+                          description: operator represents a key's relationship to
+                            a set of values. Valid operators are In, NotIn, Exists
+                            and DoesNotExist.
+                          type: string
+                        values:
+                          description: values is an array of string values. If the
+                            operator is In or NotIn, the values array must be non-empty.
+                            If the operator is Exists or DoesNotExist, the values
+                            array must be empty. This array is replaced during a strategic
+                            merge patch.
+                          items:
                             type: string
-                          fieldPath:
-                            type: string
-                          key:
-                            type: string
-                          kind:
-                            type: string
-                          name:
-                            type: string
-                          namespace:
-                            type: string
-                          resourceVersion:
-                            type: string
-                          uid:
-                            type: string
-                        type: object
-                      ingressRef:
-                        properties:
-                          apiVersion:
-                            type: string
-                          fieldPath:
-                            type: string
-                          host:
-                            type: string
-                          kind:
-                            type: string
-                          name:
-                            type: string
-                          namespace:
-                            type: string
-                          path:
-                            type: string
-                          resourceVersion:
-                            type: string
-                          uid:
-                            type: string
-                        type: object
-                      secretKeyRef:
-                        properties:
-                          apiVersion:
-                            type: string
-                          fieldPath:
-                            type: string
-                          key:
-                            type: string
-                          kind:
-                            type: string
-                          name:
-                            type: string
-                          namespace:
-                            type: string
-                          resourceVersion:
-                            type: string
-                          uid:
-                            type: string
-                        type: object
-                      serviceRef:
-                        properties:
-                          apiVersion:
-                            type: string
-                          fieldPath:
-                            type: string
-                          kind:
-                            type: string
-                          name:
-                            type: string
-                          namespace:
-                            type: string
-                          path:
-                            type: string
-                          port:
-                            format: int32
-                            type: integer
-                          resourceVersion:
-                            type: string
-                          uid:
-                            type: string
-                        type: object
-                      type:
-                        type: string
+                          type: array
+                      required:
+                      - key
+                      - operator
+                      type: object
+                    type: array
+                  matchLabels:
+                    additionalProperties:
+                      type: string
+                    description: matchLabels is a map of {key,value} pairs. A single
+                      {key,value} in the matchLabels map is equivalent to an element
+                      of matchExpressions, whose key field is "key", the operator
+                      is "In", and the values array contains only "value". The requirements
+                      are ANDed.
                     type: object
                 type: object
-              type: array
-            selector:
-              type: object
-          type: object
-        status:
-          properties:
-            components:
-              items:
-                properties:
-                  group:
-                    type: string
-                  kind:
-                    type: string
-                  link:
-                    type: string
-                  name:
-                    type: string
-                  status:
-                    type: string
-                type: object
-              type: array
-            conditions:
-              items:
-                properties:
-                  lastTransitionTime:
-                    format: date-time
-                    type: string
-                  lastUpdateTime:
-                    format: date-time
-                    type: string
-                  message:
-                    type: string
-                  reason:
-                    type: string
-                  status:
-                    type: string
-                  type:
-                    type: string
-                required:
-                - type
-                - status
-                type: object
-              type: array
-            observedGeneration:
-              format: int64
-              type: integer
-          type: object
-  version: v1beta1
+            type: object
+          status:
+            description: ApplicationStatus defines controller's the observed state
+              of Application
+            properties:
+              components:
+                description: Object status array for all matching objects
+                items:
+                  description: ObjectStatus is a generic status holder for objects
+                  properties:
+                    group:
+                      description: Object group
+                      type: string
+                    kind:
+                      description: Kind of object
+                      type: string
+                    link:
+                      description: Link to object
+                      type: string
+                    name:
+                      description: Name of object
+                      type: string
+                    status:
+                      description: 'Status. Values: InProgress, Ready, Unknown'
+                      type: string
+                  type: object
+                type: array
+              componentsReady:
+                description: 'ComponentsReady: status of the components in the format
+                  ready/total'
+                type: string
+              conditions:
+                description: Conditions represents the latest state of the object
+                items:
+                  description: Condition describes the state of an object at a certain
+                    point.
+                  properties:
+                    lastTransitionTime:
+                      description: Last time the condition transitioned from one status
+                        to another.
+                      format: date-time
+                      type: string
+                    lastUpdateTime:
+                      description: Last time the condition was probed
+                      format: date-time
+                      type: string
+                    message:
+                      description: A human readable message indicating details about
+                        the transition.
+                      type: string
+                    reason:
+                      description: The reason for the condition's last transition.
+                      type: string
+                    status:
+                      description: Status of the condition, one of True, False, Unknown.
+                      type: string
+                    type:
+                      description: Type of condition.
+                      type: string
+                  required:
+                  - status
+                  - type
+                  type: object
+                type: array
+              observedGeneration:
+                description: ObservedGeneration is the most recent generation observed.
+                  It corresponds to the Object's generation, which is updated on mutation
+                  by the API Server.
+                format: int64
+                type: integer
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
 status:
   acceptedNames:
     kind: ""

config/crds/application.kubesphere.io_helmapplications.yaml

@@ -0,0 +1,101 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: helmapplications.application.kubesphere.io
+spec:
+  group: application.kubesphere.io
+  names:
+    kind: HelmApplication
+    listKind: HelmApplicationList
+    plural: helmapplications
+    shortNames:
+    - happ
+    singular: helmapplication
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.name
+      name: application name
+      type: string
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: workspace
+      type: string
+    - jsonPath: .status.state
+      name: State
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: HelmApplication is the Schema for the helmapplications API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: HelmApplicationSpec defines the desired state of HelmApplication
+            properties:
+              abstraction:
+                description: info from frontend
+                type: string
+              appHome:
+                type: string
+              attachments:
+                description: attachments id
+                items:
+                  type: string
+                type: array
+              description:
+                description: description from chart's description or frontend
+                type: string
+              icon:
+                description: The attachment id of the icon
+                type: string
+              name:
+                description: the name of the helm application
+                type: string
+            required:
+            - name
+            type: object
+          status:
+            description: HelmApplicationStatus defines the observed state of HelmApplication
+            properties:
+              latestVersion:
+                description: If this application belong to appStore, latestVersion is the the latest version of the active application version. otherwise latestVersion is the latest version of all application version
+                type: string
+              state:
+                description: 'the state of the helm application: draft, submitted, passed, rejected, suspended, active'
+                type: string
+              statusTime:
+                format: date-time
+                type: string
+              updateTime:
+                format: date-time
+                type: string
+            required:
+            - statusTime
+            - updateTime
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/application.kubesphere.io_helmapplicationversions.yaml

@@ -0,0 +1,205 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: helmapplicationversions.application.kubesphere.io
+spec:
+  group: application.kubesphere.io
+  names:
+    kind: HelmApplicationVersion
+    listKind: HelmApplicationVersionList
+    plural: helmapplicationversions
+    shortNames:
+    - happver
+    singular: helmapplicationversion
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.name
+      name: application name
+      type: string
+    - jsonPath: .status.state
+      name: State
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: HelmApplicationVersion is the Schema for the helmapplicationversions API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: HelmApplicationVersionSpec defines the desired state of HelmApplicationVersion
+            properties:
+              annotations:
+                additionalProperties:
+                  type: string
+                description: Annotations are additional mappings uninterpreted by Helm, made available for inspection by other applications.
+                type: object
+              apiVersion:
+                description: The API Version of this chart.
+                type: string
+              appVersion:
+                description: The version of the application enclosed inside of this chart.
+                type: string
+              condition:
+                description: The condition to check to enable chart
+                type: string
+              created:
+                description: chart create time
+                format: date-time
+                type: string
+              data:
+                description: raw data of chart, it will !!!NOT!!! be save to etcd
+                format: byte
+                type: string
+              dataKey:
+                description: dataKey in the storage
+                type: string
+              dependencies:
+                description: Dependencies are a list of dependencies for a chart.
+                items:
+                  description: Dependency describes a chart upon which another chart depends. Dependencies can be used to express developer intent, or to capture the state of a chart.
+                  properties:
+                    alias:
+                      description: Alias usable alias to be used for the chart
+                      type: string
+                    condition:
+                      description: A yaml path that resolves to a boolean, used for enabling/disabling charts (e.g. subchart1.enabled )
+                      type: string
+                    enabled:
+                      description: Enabled bool determines if chart should be loaded
+                      type: boolean
+                    name:
+                      description: Name is the name of the dependency. This must mach the name in the dependency's Chart.yaml.
+                      type: string
+                    repository:
+                      description: The URL to the repository. Appending `index.yaml` to this string should result in a URL that can be used to fetch the repository index.
+                      type: string
+                    tags:
+                      description: Tags can be used to group charts for enabling/disabling together
+                      items:
+                        type: string
+                      type: array
+                    version:
+                      description: Version is the version (range) of this chart. A lock file will always produce a single version, while a dependency may contain a semantic version range.
+                      type: string
+                  required:
+                  - name
+                  - repository
+                  type: object
+                type: array
+              deprecated:
+                description: Whether or not this chart is deprecated
+                type: boolean
+              description:
+                description: A one-sentence description of the chart
+                type: string
+              digest:
+                description: chart digest
+                type: string
+              home:
+                description: The URL to a relevant project page, git repo, or contact person
+                type: string
+              icon:
+                description: The URL to an icon file.
+                type: string
+              keywords:
+                description: A list of string keywords
+                items:
+                  type: string
+                type: array
+              kubeVersion:
+                description: KubeVersion is a SemVer constraint specifying the version of Kubernetes required.
+                type: string
+              maintainers:
+                description: A list of name and URL/email address combinations for the maintainer(s)
+                items:
+                  description: Maintainer describes a Chart maintainer.
+                  properties:
+                    email:
+                      description: Email is an optional email address to contact the named maintainer
+                      type: string
+                    name:
+                      description: Name is a user name or organization name
+                      type: string
+                    url:
+                      description: URL is an optional URL to an address for the named maintainer
+                      type: string
+                  type: object
+                type: array
+              name:
+                description: The name of the chart
+                type: string
+              sources:
+                description: Source is the URL to the source code of this chart
+                items:
+                  type: string
+                type: array
+              tags:
+                description: The tags to check to enable chart
+                type: string
+              type:
+                description: 'Specifies the chart type: application or library'
+                type: string
+              urls:
+                description: chart url
+                items:
+                  type: string
+                type: array
+              version:
+                description: A SemVer 2 conformant version string of the chart
+                type: string
+            type: object
+          status:
+            description: HelmApplicationVersionStatus defines the observed state of HelmApplicationVersion
+            properties:
+              audit:
+                items:
+                  properties:
+                    message:
+                      description: audit message
+                      type: string
+                    operator:
+                      description: audit operator
+                      type: string
+                    operatorType:
+                      type: string
+                    state:
+                      description: 'audit state: submitted, passed, draft, active, rejected, suspended'
+                      type: string
+                    time:
+                      description: audit time
+                      format: date-time
+                      type: string
+                  required:
+                  - time
+                  type: object
+                type: array
+              state:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/application.kubesphere.io_helmcategories.yaml

@@ -0,0 +1,76 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: helmcategories.application.kubesphere.io
+spec:
+  group: application.kubesphere.io
+  names:
+    kind: HelmCategory
+    listKind: HelmCategoryList
+    plural: helmcategories
+    shortNames:
+    - hctg
+    singular: helmcategory
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.name
+      name: name
+      type: string
+    - jsonPath: .status.total
+      name: total
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: HelmCategory is the Schema for the helmcategories API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: HelmCategorySpec defines the desired state of HelmRepo
+            properties:
+              description:
+                description: info from frontend
+                type: string
+              locale:
+                type: string
+              name:
+                description: name of the category
+                type: string
+            required:
+            - name
+            type: object
+          status:
+            properties:
+              total:
+                description: total helmapplications belong to this category
+                type: integer
+            required:
+            - total
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/application.kubesphere.io_helmreleases.yaml

@@ -0,0 +1,145 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: helmreleases.application.kubesphere.io
+spec:
+  group: application.kubesphere.io
+  names:
+    kind: HelmRelease
+    listKind: HelmReleaseList
+    plural: helmreleases
+    shortNames:
+    - hrls
+    singular: helmrelease
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.name
+      name: Release Name
+      type: string
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: Workspace
+      type: string
+    - jsonPath: .metadata.labels.kubesphere\.io/cluster
+      name: Cluster
+      type: string
+    - jsonPath: .metadata.labels.kubesphere\.io/namespace
+      name: Namespace
+      type: string
+    - jsonPath: .status.state
+      name: State
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: HelmRelease is the Schema for the helmreleases API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: HelmReleaseSpec defines the desired state of HelmRelease
+            properties:
+              appId:
+                description: id of the helmapplication
+                type: string
+              appVerId:
+                description: application version id
+                type: string
+              chartAppVer:
+                description: appVersion from Chart.yaml
+                type: string
+              chartName:
+                description: The name of the chart which will be installed.
+                type: string
+              chartVersion:
+                description: Specify the exact chart version to install. If this is not specified, the latest version is installed
+                type: string
+              description:
+                description: Message got from frontend
+                type: string
+              name:
+                description: Name of the release
+                type: string
+              repoId:
+                description: id of  the repo
+                type: string
+              values:
+                description: helm release values.yaml
+                format: byte
+                type: string
+              version:
+                description: expected release version, when this version is not equal status.version, the release need upgrade this filed should be modified when any filed of the spec modified.
+                type: integer
+            required:
+            - chartName
+            - chartVersion
+            - name
+            - version
+            type: object
+          status:
+            description: HelmReleaseStatus defines the observed state of HelmRelease
+            properties:
+              deployStatus:
+                description: deploy status list of history, which will store at most 10 state
+                items:
+                  properties:
+                    deployTime:
+                      description: deploy time, upgrade time or check status time
+                      format: date-time
+                      type: string
+                    message:
+                      description: A human readable message indicating details about why the release is in this state.
+                      type: string
+                    state:
+                      description: current state of the release
+                      type: string
+                  required:
+                  - deployTime
+                  - state
+                  type: object
+                type: array
+              lastDeployed:
+                description: last deploy time or upgrade time
+                format: date-time
+                type: string
+              lastUpdate:
+                description: last update time
+                format: date-time
+                type: string
+              message:
+                description: A human readable message indicating details about why the release is in this state.
+                type: string
+              state:
+                description: current state
+                type: string
+              version:
+                description: current release version
+                type: integer
+            required:
+            - state
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/application.kubesphere.io_helmrepos.yaml

@@ -0,0 +1,142 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: helmrepos.application.kubesphere.io
+spec:
+  group: application.kubesphere.io
+  names:
+    kind: HelmRepo
+    listKind: HelmRepoList
+    plural: helmrepos
+    shortNames:
+    - hrepo
+    singular: helmrepo
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.name
+      name: name
+      type: string
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: Workspace
+      type: string
+    - jsonPath: .spec.url
+      name: url
+      type: string
+    - jsonPath: .status.state
+      name: State
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: HelmRepo is the Schema for the helmrepoes API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: HelmRepoSpec defines the desired state of HelmRepo
+            properties:
+              credential:
+                description: helm repo credential
+                properties:
+                  accessKeyID:
+                    type: string
+                  caFile:
+                    description: verify certificates of HTTPS-enabled servers using this CA bundle
+                    type: string
+                  certFile:
+                    description: identify HTTPS client using this SSL certificate file
+                    type: string
+                  insecureSkipTLSVerify:
+                    description: skip tls certificate checks for the repository, default is ture
+                    type: boolean
+                  keyFile:
+                    description: identify HTTPS client using this SSL key file
+                    type: string
+                  password:
+                    description: chart repository password
+                    type: string
+                  secretAccessKey:
+                    type: string
+                  username:
+                    description: chart repository username
+                    type: string
+                type: object
+              description:
+                description: chart repo description from frontend
+                type: string
+              name:
+                description: name of the repo
+                type: string
+              syncPeriod:
+                description: sync period in seconds, no sync when SyncPeriod=0, the minimum SyncPeriod is 180s
+                type: integer
+              url:
+                description: helm repo url
+                type: string
+              version:
+                description: expected repo version, when this version is not equal status.version, the repo need upgrade this filed should be modified when any filed of the spec modified.
+                type: integer
+            required:
+            - name
+            - url
+            type: object
+          status:
+            description: HelmRepoStatus defines the observed state of HelmRepo
+            properties:
+              data:
+                description: repo index
+                type: string
+              lastUpdateTime:
+                description: status last update time
+                format: date-time
+                type: string
+              state:
+                description: current state of the repo, successful, failed or syncing
+                type: string
+              syncState:
+                description: sync state list of history, which will store at most 10 state
+                items:
+                  properties:
+                    message:
+                      description: A human readable message indicating details about why the repo is in this state.
+                      type: string
+                    state:
+                      description: 'last sync state, valid state are: "failed", "success", and ""'
+                      type: string
+                    syncTime:
+                      format: date-time
+                      type: string
+                  required:
+                  - syncTime
+                  type: object
+                type: array
+              version:
+                description: if status.version!=spec.Version, we need sync the repo now
+                type: integer
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/auditing.kubesphere.io_rules.yaml

@@ -0,0 +1,86 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: rules.auditing.kubesphere.io
+spec:
+  group: auditing.kubesphere.io
+  names:
+    kind: Rule
+    listKind: RuleList
+    plural: rules
+    singular: rule
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: Rule is the Schema for the rules API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: AuditRuleSpec defines the desired state of Rule
+            properties:
+              rules:
+                items:
+                  properties:
+                    alias:
+                      description: This effective When the rule type is alias
+                      type: string
+                    condition:
+                      description: Rule condition This effective When the rule type is rule
+                      type: string
+                    desc:
+                      description: Rule describe
+                      type: string
+                    enable:
+                      description: Is the rule enable
+                      type: boolean
+                    list:
+                      description: This effective When the rule type is list
+                      items:
+                        type: string
+                      type: array
+                    macro:
+                      description: This effective When the rule type is macro
+                      type: string
+                    name:
+                      description: Rule name
+                      type: string
+                    output:
+                      description: The output formater of message which send to user
+                      type: string
+                    priority:
+                      description: Rule priority, DEBUG, INFO, WARNING
+                      type: string
+                    type:
+                      description: Rule type, rule, macro,list,alias
+                      type: string
+                  required:
+                  - enable
+                  type: object
+                type: array
+            type: object
+          status:
+            description: AuditRuleStatus defines the observed state of Rule
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/auditing.kubesphere.io_webhooks.yaml

@@ -0,0 +1,583 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: webhooks.auditing.kubesphere.io
+spec:
+  group: auditing.kubesphere.io
+  names:
+    kind: Webhook
+    listKind: WebhookList
+    plural: webhooks
+    singular: webhook
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: Webhook is the Schema for the webhooks API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: WebhookSpec defines the desired state of Webhook
+            properties:
+              affinity:
+                description: If specified, the pod's scheduling constraints
+                properties:
+                  nodeAffinity:
+                    description: Describes node affinity scheduling rules for the pod.
+                    properties:
+                      preferredDuringSchedulingIgnoredDuringExecution:
+                        description: The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node matches the corresponding matchExpressions; the node(s) with the highest sum are the most preferred.
+                        items:
+                          description: An empty preferred scheduling term matches all objects with implicit weight 0 (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op).
+                          properties:
+                            preference:
+                              description: A node selector term, associated with the corresponding weight.
+                              properties:
+                                matchExpressions:
+                                  description: A list of node selector requirements by node's labels.
+                                  items:
+                                    description: A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                                    properties:
+                                      key:
+                                        description: The label key that the selector applies to.
+                                        type: string
+                                      operator:
+                                        description: Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                        type: string
+                                      values:
+                                        description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch.
+                                        items:
+                                          type: string
+                                        type: array
+                                    required:
+                                    - key
+                                    - operator
+                                    type: object
+                                  type: array
+                                matchFields:
+                                  description: A list of node selector requirements by node's fields.
+                                  items:
+                                    description: A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                                    properties:
+                                      key:
+                                        description: The label key that the selector applies to.
+                                        type: string
+                                      operator:
+                                        description: Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                        type: string
+                                      values:
+                                        description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch.
+                                        items:
+                                          type: string
+                                        type: array
+                                    required:
+                                    - key
+                                    - operator
+                                    type: object
+                                  type: array
+                              type: object
+                            weight:
+                              description: Weight associated with matching the corresponding nodeSelectorTerm, in the range 1-100.
+                              format: int32
+                              type: integer
+                          required:
+                          - preference
+                          - weight
+                          type: object
+                        type: array
+                      requiredDuringSchedulingIgnoredDuringExecution:
+                        description: If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to an update), the system may or may not try to eventually evict the pod from its node.
+                        properties:
+                          nodeSelectorTerms:
+                            description: Required. A list of node selector terms. The terms are ORed.
+                            items:
+                              description: A null or empty node selector term matches no objects. The requirements of them are ANDed. The TopologySelectorTerm type implements a subset of the NodeSelectorTerm.
+                              properties:
+                                matchExpressions:
+                                  description: A list of node selector requirements by node's labels.
+                                  items:
+                                    description: A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                                    properties:
+                                      key:
+                                        description: The label key that the selector applies to.
+                                        type: string
+                                      operator:
+                                        description: Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                        type: string
+                                      values:
+                                        description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch.
+                                        items:
+                                          type: string
+                                        type: array
+                                    required:
+                                    - key
+                                    - operator
+                                    type: object
+                                  type: array
+                                matchFields:
+                                  description: A list of node selector requirements by node's fields.
+                                  items:
+                                    description: A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                                    properties:
+                                      key:
+                                        description: The label key that the selector applies to.
+                                        type: string
+                                      operator:
+                                        description: Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
+                                        type: string
+                                      values:
+                                        description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch.
+                                        items:
+                                          type: string
+                                        type: array
+                                    required:
+                                    - key
+                                    - operator
+                                    type: object
+                                  type: array
+                              type: object
+                            type: array
+                        required:
+                        - nodeSelectorTerms
+                        type: object
+                    type: object
+                  podAffinity:
+                    description: Describes pod affinity scheduling rules (e.g. co-locate this pod in the same node, zone, etc. as some other pod(s)).
+                    properties:
+                      preferredDuringSchedulingIgnoredDuringExecution:
+                        description: The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred.
+                        items:
+                          description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s)
+                          properties:
+                            podAffinityTerm:
+                              description: Required. A pod affinity term, associated with the corresponding weight.
+                              properties:
+                                labelSelector:
+                                  description: A label query over a set of resources, in this case pods.
+                                  properties:
+                                    matchExpressions:
+                                      description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                      items:
+                                        description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                                        properties:
+                                          key:
+                                            description: key is the label key that the selector applies to.
+                                            type: string
+                                          operator:
+                                            description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                                            type: string
+                                          values:
+                                            description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                                            items:
+                                              type: string
+                                            type: array
+                                        required:
+                                        - key
+                                        - operator
+                                        type: object
+                                      type: array
+                                    matchLabels:
+                                      additionalProperties:
+                                        type: string
+                                      description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                      type: object
+                                  type: object
+                                namespaces:
+                                  description: namespaces specifies which namespaces the labelSelector applies to (matches against); null or empty list means "this pod's namespace"
+                                  items:
+                                    type: string
+                                  type: array
+                                topologyKey:
+                                  description: This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed.
+                                  type: string
+                              required:
+                              - topologyKey
+                              type: object
+                            weight:
+                              description: weight associated with matching the corresponding podAffinityTerm, in the range 1-100.
+                              format: int32
+                              type: integer
+                          required:
+                          - podAffinityTerm
+                          - weight
+                          type: object
+                        type: array
+                      requiredDuringSchedulingIgnoredDuringExecution:
+                        description: If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied.
+                        items:
+                          description: Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key <topologyKey> matches that of any node on which a pod of the set of pods is running
+                          properties:
+                            labelSelector:
+                              description: A label query over a set of resources, in this case pods.
+                              properties:
+                                matchExpressions:
+                                  description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                  items:
+                                    description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                                    properties:
+                                      key:
+                                        description: key is the label key that the selector applies to.
+                                        type: string
+                                      operator:
+                                        description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                                        type: string
+                                      values:
+                                        description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                                        items:
+                                          type: string
+                                        type: array
+                                    required:
+                                    - key
+                                    - operator
+                                    type: object
+                                  type: array
+                                matchLabels:
+                                  additionalProperties:
+                                    type: string
+                                  description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                  type: object
+                              type: object
+                            namespaces:
+                              description: namespaces specifies which namespaces the labelSelector applies to (matches against); null or empty list means "this pod's namespace"
+                              items:
+                                type: string
+                              type: array
+                            topologyKey:
+                              description: This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed.
+                              type: string
+                          required:
+                          - topologyKey
+                          type: object
+                        type: array
+                    type: object
+                  podAntiAffinity:
+                    description: Describes pod anti-affinity scheduling rules (e.g. avoid putting this pod in the same node, zone, etc. as some other pod(s)).
+                    properties:
+                      preferredDuringSchedulingIgnoredDuringExecution:
+                        description: The scheduler will prefer to schedule pods to nodes that satisfy the anti-affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling anti-affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred.
+                        items:
+                          description: The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s)
+                          properties:
+                            podAffinityTerm:
+                              description: Required. A pod affinity term, associated with the corresponding weight.
+                              properties:
+                                labelSelector:
+                                  description: A label query over a set of resources, in this case pods.
+                                  properties:
+                                    matchExpressions:
+                                      description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                      items:
+                                        description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                                        properties:
+                                          key:
+                                            description: key is the label key that the selector applies to.
+                                            type: string
+                                          operator:
+                                            description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                                            type: string
+                                          values:
+                                            description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                                            items:
+                                              type: string
+                                            type: array
+                                        required:
+                                        - key
+                                        - operator
+                                        type: object
+                                      type: array
+                                    matchLabels:
+                                      additionalProperties:
+                                        type: string
+                                      description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                      type: object
+                                  type: object
+                                namespaces:
+                                  description: namespaces specifies which namespaces the labelSelector applies to (matches against); null or empty list means "this pod's namespace"
+                                  items:
+                                    type: string
+                                  type: array
+                                topologyKey:
+                                  description: This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed.
+                                  type: string
+                              required:
+                              - topologyKey
+                              type: object
+                            weight:
+                              description: weight associated with matching the corresponding podAffinityTerm, in the range 1-100.
+                              format: int32
+                              type: integer
+                          required:
+                          - podAffinityTerm
+                          - weight
+                          type: object
+                        type: array
+                      requiredDuringSchedulingIgnoredDuringExecution:
+                        description: If the anti-affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the anti-affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied.
+                        items:
+                          description: Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key <topologyKey> matches that of any node on which a pod of the set of pods is running
+                          properties:
+                            labelSelector:
+                              description: A label query over a set of resources, in this case pods.
+                              properties:
+                                matchExpressions:
+                                  description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                                  items:
+                                    description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                                    properties:
+                                      key:
+                                        description: key is the label key that the selector applies to.
+                                        type: string
+                                      operator:
+                                        description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                                        type: string
+                                      values:
+                                        description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                                        items:
+                                          type: string
+                                        type: array
+                                    required:
+                                    - key
+                                    - operator
+                                    type: object
+                                  type: array
+                                matchLabels:
+                                  additionalProperties:
+                                    type: string
+                                  description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                                  type: object
+                              type: object
+                            namespaces:
+                              description: namespaces specifies which namespaces the labelSelector applies to (matches against); null or empty list means "this pod's namespace"
+                              items:
+                                type: string
+                              type: array
+                            topologyKey:
+                              description: This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed.
+                              type: string
+                          required:
+                          - topologyKey
+                          type: object
+                        type: array
+                    type: object
+                type: object
+              args:
+                description: Arguments to the entrypoint.. It will be appended to the args and replace the default value.
+                items:
+                  type: string
+                type: array
+              auditLevel:
+                description: 'The Level that all requests are recorded at. available options: None, Metadata, Request, RequestResponse default: Metadata'
+                type: string
+              auditSinkPolicy:
+                description: AuditSinkPolicy is a rule selector, only the rule matched this selector will be taked effect.
+                properties:
+                  alertingRuleSelector:
+                    description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  archivingRuleSelector:
+                    description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                type: object
+              auditType:
+                description: Audit type, static or dynamic.
+                type: string
+              image:
+                description: The webhook docker image name.
+                type: string
+              imagePullPolicy:
+                description: 'Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images'
+                type: string
+              imagePullSecrets:
+                description: 'ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod'
+                items:
+                  description: LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
+                  properties:
+                    name:
+                      description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?'
+                      type: string
+                  type: object
+                type: array
+              k8sAuditingEnabled:
+                description: K8s auditing is enabled or not.
+                type: boolean
+              nodeSelector:
+                additionalProperties:
+                  type: string
+                description: 'NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node''s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/'
+                type: object
+              priority:
+                description: Rule priority, DEBUG < INFO < WARNING Audit events will be stored only when the priority of the audit rule matching the audit event is greater than this.
+                type: string
+              receivers:
+                description: ' Receiver contains the information to make a connection with the alertmanager'
+                items:
+                  description: Receiver config which received the audit alert
+                  properties:
+                    config:
+                      description: ClientConfig holds the connection parameters for the webhook
+                      properties:
+                        caBundle:
+                          description: '`caBundle` is a PEM encoded CA bundle which will be used to validate the webhook''s server certificate. If unspecified, system trust roots on the apiserver are used.'
+                          format: byte
+                          type: string
+                        service:
+                          description: "`service` is a reference to the service for this webhook. Either `service` or `url` must be specified. \n If the webhook is running within the cluster, then you should use `service`."
+                          properties:
+                            name:
+                              description: '`name` is the name of the service. Required'
+                              type: string
+                            namespace:
+                              description: '`namespace` is the namespace of the service. Required'
+                              type: string
+                            path:
+                              description: '`path` is an optional URL path which will be sent in any request to this service.'
+                              type: string
+                            port:
+                              description: If specified, the port on the service that hosting webhook. Default to 443 for backward compatibility. `port` should be a valid port number (1-65535, inclusive).
+                              format: int32
+                              type: integer
+                          required:
+                          - name
+                          - namespace
+                          type: object
+                        url:
+                          description: "`url` gives the location of the webhook, in standard URL form (`scheme://host:port/path`). Exactly one of `url` or `service` must be specified. \n The `host` should not refer to a service running in the cluster; use the `service` field instead. The host might be resolved via external DNS in some apiservers (e.g., `kube-apiserver` cannot resolve in-cluster DNS as that would be a layering violation). `host` may also be an IP address. \n Please note that using `localhost` or `127.0.0.1` as a `host` is risky unless you take great care to run this webhook on all hosts which run an apiserver which might need to make calls to this webhook. Such installs are likely to be non-portable, i.e., not easy to turn up in a new cluster. \n The scheme must be \"https\"; the URL must begin with \"https://\". \n A path is optional, and if present may be any string permissible in a URL. You may use the path to pass an arbitrary string to the webhook, for example, a cluster identifier. \n Attempting to use a user or basic auth e.g. \"user:password@\" is not allowed. Fragments (\"#...\") and query parameters (\"?...\") are not allowed, either."
+                          type: string
+                      type: object
+                    name:
+                      description: Receiver name
+                      type: string
+                    type:
+                      description: Receiver type, alertmanager or webhook
+                      type: string
+                  type: object
+                type: array
+              replicas:
+                description: Number of desired pods. This is a pointer to distinguish between explicit zero and not specified. Defaults to 1.
+                format: int32
+                type: integer
+              resources:
+                description: 'Compute Resources required by this container. Cannot be updated. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
+                properties:
+                  limits:
+                    additionalProperties:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                    description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
+                    type: object
+                  requests:
+                    additionalProperties:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                    description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
+                    type: object
+                type: object
+              tolerations:
+                description: If specified, the pod's tolerations.
+                items:
+                  description: The pod this Toleration is attached to tolerates any taint that matches the triple <key,value,effect> using the matching operator <operator>.
+                  properties:
+                    effect:
+                      description: Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.
+                      type: string
+                    key:
+                      description: Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys.
+                      type: string
+                    operator:
+                      description: Operator represents a key's relationship to the value. Valid operators are Exists and Equal. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category.
+                      type: string
+                    tolerationSeconds:
+                      description: TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.
+                      format: int64
+                      type: integer
+                    value:
+                      description: Value is the taint value the toleration matches to. If the operator is Exists, the value should be empty, otherwise just a regular string.
+                      type: string
+                  type: object
+                type: array
+            type: object
+          status:
+            description: WebhookStatus defines the observed state of Webhook
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/cluster.kubesphere.io_clusters.yaml

@@ -0,0 +1,146 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: clusters.cluster.kubesphere.io
+spec:
+  group: cluster.kubesphere.io
+  names:
+    kind: Cluster
+    listKind: ClusterList
+    plural: clusters
+    singular: cluster
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.joinFederation
+      name: Federated
+      type: boolean
+    - jsonPath: .spec.provider
+      name: Provider
+      type: string
+    - jsonPath: .spec.enable
+      name: Active
+      type: boolean
+    - jsonPath: .status.kubernetesVersion
+      name: Version
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: Cluster is the schema for the clusters API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              connection:
+                description: Connection holds info to connect to the member cluster
+                properties:
+                  kubeconfig:
+                    description: KubeConfig content used to connect to cluster api server Should provide this field explicitly if connection type is direct. Will be populated by ks-proxy if connection type is proxy.
+                    format: byte
+                    type: string
+                  kubernetesAPIEndpoint:
+                    description: 'Kubernetes API Server endpoint. Example: https://10.10.0.1:6443 Should provide this field explicitly if connection type is direct. Will be populated by ks-apiserver if connection type is proxy.'
+                    type: string
+                  kubernetesAPIServerPort:
+                    description: KubeAPIServerPort is the port which listens for forwarding kube-apiserver traffic Only applicable when connection type is proxy.
+                    type: integer
+                  kubesphereAPIEndpoint:
+                    description: 'KubeSphere API Server endpoint. Example: http://10.10.0.11:8080 Should provide this field explicitly if connection type is direct. Will be populated by ks-apiserver if connection type is proxy.'
+                    type: string
+                  kubesphereAPIServerPort:
+                    description: KubeSphereAPIServerPort is the port which listens for forwarding kubesphere apigateway traffic Only applicable when connection type is proxy.
+                    type: integer
+                  token:
+                    description: Token used by agents of member cluster to connect to host cluster proxy. This field is populated by apiserver only if connection type is proxy.
+                    type: string
+                  type:
+                    description: type defines how host cluster will connect to host cluster ConnectionTypeDirect means direct connection, this requires   kubeconfig and kubesphere apiserver endpoint provided ConnectionTypeProxy means using kubesphere proxy, no kubeconfig   or kubesphere apiserver endpoint required
+                    type: string
+                type: object
+              enable:
+                description: Desired state of the cluster
+                type: boolean
+              joinFederation:
+                description: Join cluster as a kubefed cluster
+                type: boolean
+              provider:
+                description: Provider of the cluster, this field is just for description
+                type: string
+            type: object
+          status:
+            properties:
+              conditions:
+                description: Represents the latest available observations of a cluster's current state.
+                items:
+                  properties:
+                    lastTransitionTime:
+                      description: Last time the condition transitioned from one status to another.
+                      format: date-time
+                      type: string
+                    lastUpdateTime:
+                      description: The last time this condition was updated.
+                      format: date-time
+                      type: string
+                    message:
+                      description: A human readable message indicating details about the transition.
+                      type: string
+                    reason:
+                      description: The reason for the condition's last transition.
+                      type: string
+                    status:
+                      description: Status of the condition, one of True, False, Unknown.
+                      type: string
+                    type:
+                      description: Type of the condition
+                      type: string
+                  required:
+                  - status
+                  - type
+                  type: object
+                type: array
+              configz:
+                additionalProperties:
+                  type: boolean
+                description: Configz is status of components enabled in the member cluster. This is synchronized with member cluster every amount of time, like 5 minutes.
+                type: object
+              kubeSphereVersion:
+                description: GitVersion of the /kapis/version api response, this field is populated by cluster controller
+                type: string
+              kubernetesVersion:
+                description: GitVersion of the kubernetes cluster, this field is populated by cluster controller
+                type: string
+              nodeCount:
+                description: Count of the kubernetes cluster nodes This field may not reflect the instant status of the cluster.
+                type: integer
+              region:
+                description: Region is the name of the region in which all of the nodes in the cluster exist.  e.g. 'us-east1'.
+                type: string
+              zones:
+                description: Zones are the names of availability zones in which the nodes of the cluster exist, e.g. 'us-east1-a'.
+                items:
+                  type: string
+                type: array
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/crd.projectcalico.org_blockaffinities.yaml

@@ -0,0 +1,56 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: blockaffinities.crd.projectcalico.org
+spec:
+  group: crd.projectcalico.org
+  names:
+    kind: BlockAffinity
+    listKind: BlockAffinityList
+    plural: blockaffinities
+    singular: blockaffinity
+  scope: Cluster
+  versions:
+  - name: calicov3
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: BlockAffinitySpec contains the specification for a BlockAffinity resource.
+            properties:
+              cidr:
+                type: string
+              deleted:
+                type: string
+              node:
+                type: string
+              state:
+                type: string
+            required:
+            - cidr
+            - deleted
+            - node
+            - state
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/crd.projectcalico.org_ipamblocks.yaml

@@ -0,0 +1,81 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: ipamblocks.crd.projectcalico.org
+spec:
+  group: crd.projectcalico.org
+  names:
+    kind: IPAMBlock
+    listKind: IPAMBlockList
+    plural: ipamblocks
+    singular: ipamblock
+  scope: Cluster
+  versions:
+  - name: calicov3
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: IPAMBlockSpec contains the specification for a IPAMBlock resource.
+            properties:
+              affinity:
+                type: string
+              allocations:
+                items:
+                  type: integer
+                type: array
+              attributes:
+                items:
+                  properties:
+                    handle_id:
+                      type: string
+                    secondary:
+                      additionalProperties:
+                        type: string
+                      type: object
+                  required:
+                  - handle_id
+                  - secondary
+                  type: object
+                type: array
+              cidr:
+                type: string
+              deleted:
+                type: boolean
+              strictAffinity:
+                type: boolean
+              unallocated:
+                items:
+                  type: integer
+                type: array
+            required:
+            - affinity
+            - allocations
+            - attributes
+            - cidr
+            - deleted
+            - strictAffinity
+            - unallocated
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/crd.projectcalico.org_ippools.yaml

@@ -0,0 +1,79 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: ippools.crd.projectcalico.org
+spec:
+  group: crd.projectcalico.org
+  names:
+    kind: IPPool
+    listKind: IPPoolList
+    plural: ippools
+    singular: ippool
+  scope: Cluster
+  versions:
+  - name: calicov3
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: IPPoolSpec contains the specification for an IPPool resource.
+            properties:
+              blockSize:
+                description: The block size to use for IP address assignments from this pool. Defaults to 26 for IPv4 and 112 for IPv6.
+                type: integer
+              cidr:
+                description: The pool CIDR.
+                type: string
+              disabled:
+                description: When disabled is true, Calico IPAM will not assign addresses from this pool.
+                type: boolean
+              ipip:
+                description: 'Deprecated: this field is only used for APIv1 backwards compatibility. Setting this field is not allowed, this field is for internal use only.'
+                properties:
+                  enabled:
+                    description: When enabled is true, ipip tunneling will be used to deliver packets to destinations within this pool.
+                    type: boolean
+                  mode:
+                    description: The IPIP mode.  This can be one of "always" or "cross-subnet".  A mode of "always" will also use IPIP tunneling for routing to destination IP addresses within this pool.  A mode of "cross-subnet" will only use IPIP tunneling when the destination node is on a different subnet to the originating node.  The default value (if not specified) is "always".
+                    type: string
+                type: object
+              ipipMode:
+                description: Contains configuration for IPIP tunneling for this pool. If not specified, then this is defaulted to "Never" (i.e. IPIP tunelling is disabled).
+                type: string
+              nat-outgoing:
+                description: 'Deprecated: this field is only used for APIv1 backwards compatibility. Setting this field is not allowed, this field is for internal use only.'
+                type: boolean
+              natOutgoing:
+                description: When nat-outgoing is true, packets sent from Calico networked containers in this pool to destinations outside of this pool will be masqueraded.
+                type: boolean
+              nodeSelector:
+                description: Allows IPPool to allocate for a specific node by label selector.
+                type: string
+              vxlanMode:
+                description: Contains configuration for VXLAN tunneling for this pool. If not specified, then this is defaulted to "Never" (i.e. VXLAN tunelling is disabled).
+                type: string
+            required:
+            - cidr
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_devopsprojects.yaml

@@ -0,0 +1,52 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: devopsprojects.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    categories:
+    - devops
+    kind: DevOpsProject
+    listKind: DevOpsProjectList
+    plural: devopsprojects
+    singular: devopsproject
+  scope: Cluster
+  versions:
+  - name: v1alpha3
+    schema:
+      openAPIV3Schema:
+        description: DevOpsProject is the Schema for the devopsprojects API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: DevOpsProjectSpec defines the desired state of DevOpsProject
+            type: object
+          status:
+            description: DevOpsProjectStatus defines the observed state of DevOpsProject
+            properties:
+              adminNamespace:
+                description: 'INSERT ADDITIONAL STATUS FIELD - define observed state of cluster Important: Run "make" to regenerate code after modifying this file'
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_pipelines.yaml

@@ -0,0 +1,300 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: pipelines.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    kind: Pipeline
+    listKind: PipelineList
+    plural: pipelines
+    singular: pipeline
+  scope: Namespaced
+  versions:
+  - name: v1alpha3
+    schema:
+      openAPIV3Schema:
+        description: Pipeline is the Schema for the pipelines API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: PipelineSpec defines the desired state of Pipeline
+            properties:
+              multi_branch_pipeline:
+                properties:
+                  bitbucket_server_source:
+                    properties:
+                      api_uri:
+                        type: string
+                      credential_id:
+                        type: string
+                      discover_branches:
+                        type: integer
+                      discover_pr_from_forks:
+                        properties:
+                          strategy:
+                            type: integer
+                          trust:
+                            type: integer
+                        type: object
+                      discover_pr_from_origin:
+                        type: integer
+                      discover_tags:
+                        type: boolean
+                      git_clone_option:
+                        properties:
+                          depth:
+                            type: integer
+                          shallow:
+                            type: boolean
+                          timeout:
+                            type: integer
+                        type: object
+                      owner:
+                        type: string
+                      regex_filter:
+                        type: string
+                      repo:
+                        type: string
+                      scm_id:
+                        type: string
+                    type: object
+                  description:
+                    type: string
+                  discarder:
+                    properties:
+                      days_to_keep:
+                        type: string
+                      num_to_keep:
+                        type: string
+                    type: object
+                  git_source:
+                    properties:
+                      credential_id:
+                        type: string
+                      discover_branches:
+                        type: boolean
+                      discover_tags:
+                        type: boolean
+                      git_clone_option:
+                        properties:
+                          depth:
+                            type: integer
+                          shallow:
+                            type: boolean
+                          timeout:
+                            type: integer
+                        type: object
+                      regex_filter:
+                        type: string
+                      scm_id:
+                        type: string
+                      url:
+                        type: string
+                    type: object
+                  github_source:
+                    description: GithubSource and BitbucketServerSource have the same structure, but we don't use one due to crd errors
+                    properties:
+                      api_uri:
+                        type: string
+                      credential_id:
+                        type: string
+                      discover_branches:
+                        type: integer
+                      discover_pr_from_forks:
+                        properties:
+                          strategy:
+                            type: integer
+                          trust:
+                            type: integer
+                        type: object
+                      discover_pr_from_origin:
+                        type: integer
+                      discover_tags:
+                        type: boolean
+                      git_clone_option:
+                        properties:
+                          depth:
+                            type: integer
+                          shallow:
+                            type: boolean
+                          timeout:
+                            type: integer
+                        type: object
+                      owner:
+                        type: string
+                      regex_filter:
+                        type: string
+                      repo:
+                        type: string
+                      scm_id:
+                        type: string
+                    type: object
+                  gitlab_source:
+                    properties:
+                      api_uri:
+                        type: string
+                      credential_id:
+                        type: string
+                      discover_branches:
+                        type: integer
+                      discover_pr_from_forks:
+                        properties:
+                          strategy:
+                            type: integer
+                          trust:
+                            type: integer
+                        type: object
+                      discover_pr_from_origin:
+                        type: integer
+                      discover_tags:
+                        type: boolean
+                      git_clone_option:
+                        properties:
+                          depth:
+                            type: integer
+                          shallow:
+                            type: boolean
+                          timeout:
+                            type: integer
+                        type: object
+                      owner:
+                        type: string
+                      regex_filter:
+                        type: string
+                      repo:
+                        type: string
+                      scm_id:
+                        type: string
+                      server_name:
+                        type: string
+                    type: object
+                  multibranch_job_trigger:
+                    properties:
+                      create_action_job_to_trigger:
+                        type: string
+                      delete_action_job_to_trigger:
+                        type: string
+                    type: object
+                  name:
+                    type: string
+                  script_path:
+                    type: string
+                  single_svn_source:
+                    properties:
+                      credential_id:
+                        type: string
+                      remote:
+                        type: string
+                      scm_id:
+                        type: string
+                    type: object
+                  source_type:
+                    type: string
+                  svn_source:
+                    properties:
+                      credential_id:
+                        type: string
+                      excludes:
+                        type: string
+                      includes:
+                        type: string
+                      remote:
+                        type: string
+                      scm_id:
+                        type: string
+                    type: object
+                  timer_trigger:
+                    properties:
+                      cron:
+                        description: user in no scm job
+                        type: string
+                      interval:
+                        description: use in multi-branch job
+                        type: string
+                    type: object
+                required:
+                - name
+                - script_path
+                - source_type
+                type: object
+              pipeline:
+                properties:
+                  description:
+                    type: string
+                  disable_concurrent:
+                    type: boolean
+                  discarder:
+                    properties:
+                      days_to_keep:
+                        type: string
+                      num_to_keep:
+                        type: string
+                    type: object
+                  jenkinsfile:
+                    type: string
+                  name:
+                    type: string
+                  parameters:
+                    items:
+                      properties:
+                        default_value:
+                          type: string
+                        description:
+                          type: string
+                        name:
+                          type: string
+                        type:
+                          type: string
+                      required:
+                      - name
+                      - type
+                      type: object
+                    type: array
+                  remote_trigger:
+                    properties:
+                      token:
+                        type: string
+                    type: object
+                  timer_trigger:
+                    properties:
+                      cron:
+                        description: user in no scm job
+                        type: string
+                      interval:
+                        description: use in multi-branch job
+                        type: string
+                    type: object
+                required:
+                - name
+                type: object
+              type:
+                description: 'INSERT ADDITIONAL SPEC FIELDS - desired state of cluster Important: Run "make" to regenerate code after modifying this file'
+                type: string
+            required:
+            - type
+            type: object
+          status:
+            description: PipelineStatus defines the observed state of Pipeline
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_s2ibinaries.yaml

@@ -0,0 +1,81 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: s2ibinaries.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    kind: S2iBinary
+    listKind: S2iBinaryList
+    plural: s2ibinaries
+    singular: s2ibinary
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.fileName
+      name: FileName
+      type: string
+    - jsonPath: .spec.md5
+      name: MD5
+      type: string
+    - jsonPath: .spec.size
+      name: Size
+      type: string
+    - jsonPath: .status.phase
+      name: Phase
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: S2iBinary is the Schema for the s2ibinaries API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: S2iBinarySpec defines the desired state of S2iBinary
+            properties:
+              downloadURL:
+                description: DownloadURL in KubeSphere
+                type: string
+              fileName:
+                description: FileName is filename of binary
+                type: string
+              md5:
+                description: MD5 is Binary's MD5 Hash
+                type: string
+              size:
+                description: Size is the file size of file
+                type: string
+              uploadTimeStamp:
+                description: UploadTime is last upload time
+                format: date-time
+                type: string
+            type: object
+          status:
+            description: S2iBinaryStatus defines the observed state of S2iBinary
+            properties:
+              phase:
+                description: Phase is status of S2iBinary . Possible value is "Ready","UnableToDownload"
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_s2ibuilders.yaml

@@ -0,0 +1,475 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: s2ibuilders.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    kind: S2iBuilder
+    listKind: S2iBuilderList
+    plural: s2ibuilders
+    shortNames:
+    - s2ib
+    singular: s2ibuilder
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .status.runCount
+      name: RunCount
+      type: integer
+    - jsonPath: .status.lastRunState
+      name: LastRunState
+      type: string
+    - jsonPath: .status.lastRunName
+      name: LastRunName
+      type: string
+    - jsonPath: .status.lastRunStartTime
+      name: LastRunStartTime
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: S2iBuilder is the Schema for the s2ibuilders API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: S2iBuilderSpec defines the desired state of S2iBuilder
+            properties:
+              config:
+                description: 'INSERT ADDITIONAL SPEC FIELDS - desired state of cluster Important: Run "make" to regenerate code after modifying this file'
+                properties:
+                  addHost:
+                    description: AddHost Add a line to /etc/hosts for test purpose or private use in LAN. Its format is host:IP,muliple hosts can be added  by using multiple --add-host
+                    items:
+                      type: string
+                    type: array
+                  asDockerfile:
+                    description: AsDockerfile indicates the path where the Dockerfile should be written instead of building a new image.
+                    type: string
+                  assembleUser:
+                    description: AssembleUser specifies the user to run the assemble script in container
+                    type: string
+                  blockOnBuild:
+                    description: BlockOnBuild prevents s2i from performing a docker build operation if one is necessary to execute ONBUILD commands, or to layer source code into the container for images that don't have a tar binary available, if the image contains ONBUILD commands that would be executed.
+                    type: boolean
+                  branchExpression:
+                    description: Regular expressions, ignoring names that do not match the provided regular expression
+                    type: string
+                  buildVolumes:
+                    description: BuildVolumes specifies a list of volumes to mount to container running the build.
+                    items:
+                      type: string
+                    type: array
+                  builderBaseImageVersion:
+                    description: BuilderBaseImageVersion provides optional version information about the builder base image.
+                    type: string
+                  builderImage:
+                    description: BuilderImage describes which image is used for building the result images.
+                    type: string
+                  builderImageVersion:
+                    description: BuilderImageVersion provides optional version information about the builder image.
+                    type: string
+                  builderPullPolicy:
+                    description: BuilderPullPolicy specifies when to pull the builder image
+                    type: string
+                  callbackUrl:
+                    description: CallbackURL is a URL which is called upon successful build to inform about that fact.
+                    type: string
+                  cgroupLimits:
+                    description: CGroupLimits describes the cgroups limits that will be applied to any containers run by s2i.
+                    properties:
+                      cpuPeriod:
+                        format: int64
+                        type: integer
+                      cpuQuota:
+                        format: int64
+                        type: integer
+                      cpuShares:
+                        format: int64
+                        type: integer
+                      memoryLimitBytes:
+                        format: int64
+                        type: integer
+                      memorySwap:
+                        format: int64
+                        type: integer
+                      parent:
+                        type: string
+                    required:
+                    - cpuPeriod
+                    - cpuQuota
+                    - cpuShares
+                    - memoryLimitBytes
+                    - memorySwap
+                    - parent
+                    type: object
+                  contextDir:
+                    description: Specify a relative directory inside the application repository that should be used as a root directory for the application.
+                    type: string
+                  description:
+                    description: Description is a result image description label. The default is no description.
+                    type: string
+                  destination:
+                    description: Destination specifies a location where the untar operation will place its artifacts.
+                    type: string
+                  displayName:
+                    description: DisplayName is a result image display-name label. This defaults to the output image name.
+                    type: string
+                  dockerConfig:
+                    description: DockerConfig describes how to access host docker daemon.
+                    properties:
+                      caFile:
+                        description: CAFile is the certificate authority file path for a TLS connection
+                        type: string
+                      certFile:
+                        description: CertFile is the certificate file path for a TLS connection
+                        type: string
+                      endPoint:
+                        description: Endpoint is the docker network endpoint or socket
+                        type: string
+                      keyFile:
+                        description: KeyFile is the key file path for a TLS connection
+                        type: string
+                      tlsVerify:
+                        description: TLSVerify indicates if TLS peer must be verified
+                        type: boolean
+                      useTLS:
+                        description: UseTLS indicates if TLS must be used
+                        type: boolean
+                    required:
+                    - caFile
+                    - certFile
+                    - endPoint
+                    - keyFile
+                    - tlsVerify
+                    - useTLS
+                    type: object
+                  dockerNetworkMode:
+                    description: DockerNetworkMode is used to set the docker network setting to --net=container:<id> when the builder is invoked from a container.
+                    type: string
+                  dropCapabilities:
+                    description: DropCapabilities contains a list of capabilities to drop when executing containers
+                    items:
+                      type: string
+                    type: array
+                  environment:
+                    description: Environment is a map of environment variables to be passed to the image.
+                    items:
+                      description: EnvironmentSpec specifies a single environment variable.
+                      properties:
+                        name:
+                          type: string
+                        value:
+                          type: string
+                      required:
+                      - name
+                      - value
+                      type: object
+                    type: array
+                  excludeRegExp:
+                    description: ExcludeRegExp contains a string representation of the regular expression desired for deciding which files to exclude from the tar stream
+                    type: string
+                  export:
+                    description: Export Push the result image to specify image registry in tag
+                    type: boolean
+                  gitSecretRef:
+                    description: GitSecretRef is the BasicAuth Secret of Git Clone
+                    properties:
+                      name:
+                        description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?'
+                        type: string
+                    type: object
+                  hasOnBuild:
+                    description: HasOnBuild will be set to true if the builder image contains ONBUILD instructions
+                    type: boolean
+                  imageName:
+                    description: ImageName Contains the registry address and reponame, tag should set by field tag alone
+                    type: string
+                  imageScriptsUrl:
+                    description: ImageScriptsURL is the default location to find the assemble/run scripts for a builder image. This url can be a reference within the builder image if the scheme is specified as image://
+                    type: string
+                  imageWorkDir:
+                    description: ImageWorkDir is the default working directory for the builder image.
+                    type: string
+                  incremental:
+                    description: Incremental describes whether to try to perform incremental build.
+                    type: boolean
+                  incrementalAuthentication:
+                    description: IncrementalAuthentication holds the authentication information for pulling the previous image from private repositories
+                    properties:
+                      email:
+                        type: string
+                      password:
+                        type: string
+                      secretRef:
+                        description: LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
+                        properties:
+                          name:
+                            description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?'
+                            type: string
+                        type: object
+                      serverAddress:
+                        type: string
+                      username:
+                        type: string
+                    type: object
+                  incrementalFromTag:
+                    description: IncrementalFromTag sets an alternative image tag to look for existing artifacts. Tag is used by default if this is not set.
+                    type: string
+                  injections:
+                    description: Injections specifies a list source/destination folders that are injected to the container that runs assemble. All files we inject will be truncated after the assemble script finishes.
+                    items:
+                      description: VolumeSpec represents a single volume mount point.
+                      properties:
+                        destination:
+                          description: Destination is the path to mount the volume to - absolute or relative.
+                          type: string
+                        keep:
+                          description: Keep indicates if the mounted data should be kept in the final image.
+                          type: boolean
+                        source:
+                          description: Source is a reference to the volume source.
+                          type: string
+                      type: object
+                    type: array
+                  isBinaryURL:
+                    description: IsBinaryURL explain the type of SourceURL. If it is IsBinaryURL, it will download the file directly without using git.
+                    type: boolean
+                  keepSymlinks:
+                    description: KeepSymlinks indicates to copy symlinks as symlinks. Default behavior is to follow symlinks and copy files by content.
+                    type: boolean
+                  labelNamespace:
+                    description: LabelNamespace provides the namespace under which the labels will be generated.
+                    type: string
+                  labels:
+                    additionalProperties:
+                      type: string
+                    description: Labels specify labels and their values to be applied to the resulting image. Label keys must have non-zero length. The labels defined here override generated labels in case they have the same name.
+                    type: object
+                  layeredBuild:
+                    description: LayeredBuild describes if this is build which layered scripts and sources on top of BuilderImage.
+                    type: boolean
+                  nodeAffinityKey:
+                    description: The key of Node Affinity.
+                    type: string
+                  nodeAffinityValues:
+                    description: The values of Node Affinity.
+                    items:
+                      type: string
+                    type: array
+                  outputBuildResult:
+                    description: Whether output build result to status.
+                    type: boolean
+                  outputImageName:
+                    description: OutputImageName is a result image name without tag, default is latest. tag will append to ImageName in the end
+                    type: string
+                  preserveWorkingDir:
+                    description: PreserveWorkingDir describes if working directory should be left after processing.
+                    type: boolean
+                  previousImagePullPolicy:
+                    description: PreviousImagePullPolicy specifies when to pull the previously build image when doing incremental build
+                    type: string
+                  pullAuthentication:
+                    description: PullAuthentication holds the authentication information for pulling the Docker images from private repositories
+                    properties:
+                      email:
+                        type: string
+                      password:
+                        type: string
+                      secretRef:
+                        description: LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
+                        properties:
+                          name:
+                            description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?'
+                            type: string
+                        type: object
+                      serverAddress:
+                        type: string
+                      username:
+                        type: string
+                    type: object
+                  pushAuthentication:
+                    description: PullAuthentication holds the authentication information for pulling the Docker images from private repositories
+                    properties:
+                      email:
+                        type: string
+                      password:
+                        type: string
+                      secretRef:
+                        description: LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
+                        properties:
+                          name:
+                            description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?'
+                            type: string
+                        type: object
+                      serverAddress:
+                        type: string
+                      username:
+                        type: string
+                    type: object
+                  removePreviousImage:
+                    description: RemovePreviousImage describes if previous image should be removed after successful build. This applies only to incremental builds.
+                    type: boolean
+                  revisionId:
+                    description: The RevisionId is a branch name or a SHA-1 hash of every important thing about the commit
+                    type: string
+                  runImage:
+                    description: RunImage will trigger a "docker run ..." invocation of the produced image so the user can see if it operates as he would expect
+                    type: boolean
+                  runtimeArtifacts:
+                    description: RuntimeArtifacts specifies a list of source/destination pairs that will be copied from builder to a runtime image. Source can be a file or directory. Destination must be a directory. Regardless whether it is an absolute or relative path, it will be placed into image's WORKDIR. Destination also can be empty or equals to ".", in this case it just refers to a root of WORKDIR. In case it's empty, S2I will try to get this list from io.openshift.s2i.assemble-input-files label on a RuntimeImage.
+                    items:
+                      description: VolumeSpec represents a single volume mount point.
+                      properties:
+                        destination:
+                          description: Destination is the path to mount the volume to - absolute or relative.
+                          type: string
+                        keep:
+                          description: Keep indicates if the mounted data should be kept in the final image.
+                          type: boolean
+                        source:
+                          description: Source is a reference to the volume source.
+                          type: string
+                      type: object
+                    type: array
+                  runtimeAuthentication:
+                    description: RuntimeAuthentication holds the authentication information for pulling the runtime Docker images from private repositories.
+                    properties:
+                      email:
+                        type: string
+                      password:
+                        type: string
+                      secretRef:
+                        description: LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
+                        properties:
+                          name:
+                            description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?'
+                            type: string
+                        type: object
+                      serverAddress:
+                        type: string
+                      username:
+                        type: string
+                    type: object
+                  runtimeImage:
+                    description: RuntimeImage specifies the image that will be a base for resulting image and will be used for running an application. By default, BuilderImage is used for building and running, but the latter may be overridden.
+                    type: string
+                  runtimeImagePullPolicy:
+                    description: RuntimeImagePullPolicy specifies when to pull a runtime image.
+                    type: string
+                  scriptDownloadProxyConfig:
+                    description: ScriptDownloadProxyConfig optionally specifies the http and https proxy to use when downloading scripts
+                    properties:
+                      httpProxy:
+                        type: string
+                      httpsProxy:
+                        type: string
+                    type: object
+                  scriptsUrl:
+                    description: ScriptsURL is a URL describing where to fetch the S2I scripts from during build process. This url can be a reference within the builder image if the scheme is specified as image://
+                    type: string
+                  secretCode:
+                    description: SecretCode
+                    type: string
+                  securityOpt:
+                    description: SecurityOpt are passed as options to the docker containers launched by s2i.
+                    items:
+                      type: string
+                    type: array
+                  sourceUrl:
+                    description: SourceURL is  url of the codes such as https://github.com/a/b.git
+                    type: string
+                  tag:
+                    description: Tag is a result image tag name.
+                    type: string
+                  taintKey:
+                    description: The name of taint.
+                    type: string
+                  usage:
+                    description: Usage allows for properly shortcircuiting s2i logic when `s2i usage` is invoked
+                    type: boolean
+                  workingDir:
+                    description: WorkingDir describes temporary directory used for downloading sources, scripts and tar operations.
+                    type: string
+                  workingSourceDir:
+                    description: WorkingSourceDir describes the subdirectory off of WorkingDir set up during the repo download that is later used as the root for ignore processing
+                    type: string
+                required:
+                - imageName
+                - sourceUrl
+                type: object
+              fromTemplate:
+                description: FromTemplate define some inputs from user
+                properties:
+                  builderImage:
+                    description: BaseImage specify which version of this template to use
+                    type: string
+                  name:
+                    description: Name specify a template to use, so many fields in Config can left empty
+                    type: string
+                  parameters:
+                    description: Parameters must use with `template`, fill some parameters which template will use
+                    items:
+                      properties:
+                        defaultValue:
+                          type: string
+                        description:
+                          type: string
+                        key:
+                          type: string
+                        optValues:
+                          items:
+                            type: string
+                          type: array
+                        required:
+                          type: boolean
+                        type:
+                          type: string
+                        value:
+                          type: string
+                      type: object
+                    type: array
+                type: object
+            type: object
+          status:
+            description: S2iBuilderStatus defines the observed state of S2iBuilder
+            properties:
+              lastRunName:
+                description: LastRunState return the name of the newest run of this builder
+                type: string
+              lastRunStartTime:
+                description: LastRunStartTime return the startTime of the newest run of this builder
+                format: date-time
+                type: string
+              lastRunState:
+                description: LastRunState return the state of the newest run of this builder
+                type: string
+              runCount:
+                description: RunCount represent the sum of s2irun of this builder
+                type: integer
+            required:
+            - runCount
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_s2ibuildertemplates.yaml

@@ -0,0 +1,130 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: s2ibuildertemplates.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    categories:
+    - devops
+    kind: S2iBuilderTemplate
+    listKind: S2iBuilderTemplateList
+    plural: s2ibuildertemplates
+    shortNames:
+    - s2ibt
+    singular: s2ibuildertemplate
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.codeFramework
+      name: Framework
+      type: string
+    - jsonPath: .spec.defaultBaseImage
+      name: DefaultBaseImage
+      type: string
+    - jsonPath: .spec.version
+      name: Version
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: S2iBuilderTemplate is the Schema for the s2ibuildertemplates API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: S2iBuilderTemplateSpec defines the desired state of S2iBuilderTemplate
+            properties:
+              codeFramework:
+                description: CodeFramework means which language this template is designed for and which framework is using if has framework. Like Java, NodeJS etc
+                type: string
+              containerInfo:
+                description: Images are the images this template will use.
+                items:
+                  properties:
+                    buildVolumes:
+                      description: BuildVolumes specifies a list of volumes to mount to container running the build.
+                      items:
+                        type: string
+                      type: array
+                    builderImage:
+                      description: BaseImage are the images this template will use.
+                      type: string
+                    runtimeArtifacts:
+                      items:
+                        description: VolumeSpec represents a single volume mount point.
+                        properties:
+                          destination:
+                            description: Destination is the path to mount the volume to - absolute or relative.
+                            type: string
+                          keep:
+                            description: Keep indicates if the mounted data should be kept in the final image.
+                            type: boolean
+                          source:
+                            description: Source is a reference to the volume source.
+                            type: string
+                        type: object
+                      type: array
+                    runtimeImage:
+                      type: string
+                  type: object
+                type: array
+              defaultBaseImage:
+                description: DefaultBaseImage is the image that will be used by default
+                type: string
+              description:
+                description: Description illustrate the purpose of this template
+                type: string
+              environment:
+                description: Parameters is a set of environment variables to be passed to the image.
+                items:
+                  properties:
+                    defaultValue:
+                      type: string
+                    description:
+                      type: string
+                    key:
+                      type: string
+                    optValues:
+                      items:
+                        type: string
+                      type: array
+                    required:
+                      type: boolean
+                    type:
+                      type: string
+                    value:
+                      type: string
+                  type: object
+                type: array
+              iconPath:
+                description: IconPath is used for frontend display
+                type: string
+              version:
+                description: Version of template
+                type: string
+            type: object
+          status:
+            description: S2iBuilderTemplateStatus defines the observed state of S2iBuilderTemplate
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops.kubesphere.io_s2iruns.yaml

@@ -0,0 +1,164 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: s2iruns.devops.kubesphere.io
+spec:
+  group: devops.kubesphere.io
+  names:
+    kind: S2iRun
+    listKind: S2iRunList
+    plural: s2iruns
+    shortNames:
+    - s2ir
+    singular: s2irun
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .status.runState
+      name: State
+      type: string
+    - jsonPath: .status.kubernetesJobName
+      name: K8sJobName
+      type: string
+    - jsonPath: .status.startTime
+      name: StartTime
+      type: date
+    - jsonPath: .status.completionTime
+      name: CompletionTime
+      type: date
+    - jsonPath: .status.s2iBuildResult.imageName
+      name: ImageName
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: S2iRun is the Schema for the s2iruns API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: S2iRunSpec defines the desired state of S2iRun
+            properties:
+              backoffLimit:
+                description: BackoffLimit limits the restart count of each s2irun. Default is 0
+                format: int32
+                type: integer
+              builderName:
+                description: BuilderName specify the name of s2ibuilder, required
+                type: string
+              newRevisionId:
+                description: NewRevisionId override the default NewRevisionId in its s2ibuilder.
+                type: string
+              newSourceURL:
+                description: NewSourceURL is used to download new binary artifacts
+                type: string
+              newTag:
+                description: NewTag override the default tag in its s2ibuilder, image name cannot be changed.
+                type: string
+              secondsAfterFinished:
+                description: SecondsAfterFinished if is set and greater than zero, and the job created by s2irun become successful or failed , the job will be auto deleted after SecondsAfterFinished
+                format: int32
+                type: integer
+            required:
+            - builderName
+            type: object
+          status:
+            description: S2iRunStatus defines the observed state of S2iRun
+            properties:
+              completionTime:
+                description: Represents time when the job was completed. It is not guaranteed to be set in happens-before order across separate operations. It is represented in RFC3339 form and is in UTC.
+                format: date-time
+                type: string
+              kubernetesJobName:
+                description: KubernetesJobName is the job name in k8s
+                type: string
+              logURL:
+                description: LogURL is uesd for external log handler to let user know where is log located in
+                type: string
+              runState:
+                description: RunState  indicates whether this job is done or failed
+                type: string
+              s2iBuildResult:
+                description: S2i build result info.
+                properties:
+                  commandPull:
+                    description: Command for pull image.
+                    type: string
+                  imageCreated:
+                    description: Image created time.
+                    type: string
+                  imageID:
+                    description: Image ID.
+                    type: string
+                  imageName:
+                    description: ImageName is the name of artifact
+                    type: string
+                  imageRepoTags:
+                    description: image tags.
+                    items:
+                      type: string
+                    type: array
+                  imageSize:
+                    description: The size in bytes of the image
+                    format: int64
+                    type: integer
+                type: object
+              s2iBuildSource:
+                description: S2i build source info.
+                properties:
+                  binaryName:
+                    description: Binary file Name
+                    type: string
+                  binarySize:
+                    description: Binary file Size
+                    format: int64
+                    type: integer
+                  builderImage:
+                    description: // BuilderImage describes which image is used for building the result images.
+                    type: string
+                  commitID:
+                    description: CommitID represents an arbitrary extended object reference in Git as SHA-1
+                    type: string
+                  committerEmail:
+                    description: CommitterEmail contains the e-mail of the committer
+                    type: string
+                  committerName:
+                    description: CommitterName contains the name of the committer
+                    type: string
+                  description:
+                    description: Description is a result image description label. The default is no description.
+                    type: string
+                  revisionId:
+                    description: The RevisionId is a branch name or a SHA-1 hash of every important thing about the commit
+                    type: string
+                  sourceUrl:
+                    description: SourceURL is  url of the codes such as https://github.com/a/b.git
+                    type: string
+                type: object
+              startTime:
+                description: StartTime represent when this run began
+                format: date-time
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/devops_v1alpha1_s2ibinary.yaml

@@ -1,73 +0,0 @@
-apiVersion: apiextensions.k8s.io/v1beta1
-kind: CustomResourceDefinition
-metadata:
-  creationTimestamp: null
-  labels:
-    controller-tools.k8s.io: "1.0"
-  name: s2ibinaries.devops.kubesphere.io
-spec:
-  additionalPrinterColumns:
-  - JSONPath: .spec.fileName
-    name: FileName
-    type: string
-  - JSONPath: .spec.md5
-    name: MD5
-    type: string
-  - JSONPath: .spec.size
-    name: Size
-    type: string
-  - JSONPath: .status.phase
-    name: Phase
-    type: string
-  group: devops.kubesphere.io
-  names:
-    kind: S2iBinary
-    plural: s2ibinaries
-  scope: Namespaced
-  validation:
-    openAPIV3Schema:
-      properties:
-        apiVersion:
-          description: 'APIVersion defines the versioned schema of this representation
-            of an object. Servers should convert recognized schemas to the latest
-            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
-          type: string
-        kind:
-          description: 'Kind is a string value representing the REST resource this
-            object represents. Servers may infer this from the endpoint the client
-            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
-          type: string
-        metadata:
-          type: object
-        spec:
-          properties:
-            downloadURL:
-              description: DownloadURL in KubeSphere
-              type: string
-            fileName:
-              description: FileName is filename of binary
-              type: string
-            md5:
-              description: MD5 is Binary's MD5 Hash
-              type: string
-            size:
-              description: Size is the file size of file
-              type: string
-            uploadTimeStamp:
-              description: UploadTime is last upload time
-              format: date-time
-              type: string
-          type: object
-        status:
-          properties:
-            phase:
-              description: Phase is status of S2iBinary . Possible value is "Ready","UnableToDownload"
-              type: string
-          type: object
-  version: v1alpha1
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crds/iam.kubesphere.io_federatedrolebindings.yaml

@@ -0,0 +1,111 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: federatedrolebindings.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    kind: FederatedRoleBinding
+    listKind: FederatedRoleBindingList
+    plural: federatedrolebindings
+    singular: federatedrolebinding
+  scope: Namespaced
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              placement:
+                properties:
+                  clusterSelector:
+                    properties:
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        type: object
+                    type: object
+                  clusters:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                      required:
+                      - name
+                      type: object
+                    type: array
+                type: object
+              template:
+                properties:
+                  metadata:
+                    type: object
+                  roleRef:
+                    description: RoleRef contains information that points to the role being used
+                    properties:
+                      apiGroup:
+                        description: APIGroup is the group for the resource being referenced
+                        type: string
+                      kind:
+                        description: Kind is the type of resource being referenced
+                        type: string
+                      name:
+                        description: Name is the name of resource being referenced
+                        type: string
+                    required:
+                    - apiGroup
+                    - kind
+                    - name
+                    type: object
+                  subjects:
+                    items:
+                      description: Subject contains a reference to the object or user identities a role binding applies to.  This can either hold a direct API object reference, or a value for non-objects such as user and group names.
+                      properties:
+                        apiGroup:
+                          description: APIGroup holds the API group of the referenced subject. Defaults to "" for ServiceAccount subjects. Defaults to "rbac.authorization.k8s.io" for User and Group subjects.
+                          type: string
+                        kind:
+                          description: Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount". If the Authorizer does not recognized the kind value, the Authorizer should report an error.
+                          type: string
+                        name:
+                          description: Name of the object being referenced.
+                          type: string
+                        namespace:
+                          description: Namespace of the referenced object.  If the object kind is non-namespace, such as "User" or "Group", and this value is not empty the Authorizer should report an error.
+                          type: string
+                      required:
+                      - kind
+                      - name
+                      type: object
+                    type: array
+                required:
+                - roleRef
+                type: object
+            required:
+            - placement
+            - template
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_federatedroles.yaml

@@ -0,0 +1,104 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: federatedroles.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    kind: FederatedRole
+    listKind: FederatedRoleList
+    plural: federatedroles
+    singular: federatedrole
+  scope: Namespaced
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              placement:
+                properties:
+                  clusterSelector:
+                    properties:
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        type: object
+                    type: object
+                  clusters:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                      required:
+                      - name
+                      type: object
+                    type: array
+                type: object
+              template:
+                properties:
+                  metadata:
+                    type: object
+                  rules:
+                    items:
+                      description: PolicyRule holds information that describes a policy rule, but does not contain information about who the rule applies to or which namespace the rule applies to.
+                      properties:
+                        apiGroups:
+                          description: APIGroups is the name of the APIGroup that contains the resources.  If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed.
+                          items:
+                            type: string
+                          type: array
+                        nonResourceURLs:
+                          description: NonResourceURLs is a set of partial urls that a user should have access to.  *s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"),  but not both.
+                          items:
+                            type: string
+                          type: array
+                        resourceNames:
+                          description: ResourceNames is an optional white list of names that the rule applies to.  An empty set means that everything is allowed.
+                          items:
+                            type: string
+                          type: array
+                        resources:
+                          description: Resources is a list of resources this rule applies to.  ResourceAll represents all resources.
+                          items:
+                            type: string
+                          type: array
+                        verbs:
+                          description: Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule.  VerbAll represents all kinds.
+                          items:
+                            type: string
+                          type: array
+                      required:
+                      - verbs
+                      type: object
+                    type: array
+                type: object
+            required:
+            - placement
+            - template
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_federatedusers.yaml

@@ -0,0 +1,113 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: federatedusers.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    kind: FederatedUser
+    listKind: FederatedUserList
+    plural: federatedusers
+    singular: federateduser
+  scope: Namespaced
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              placement:
+                properties:
+                  clusterSelector:
+                    properties:
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        type: object
+                    type: object
+                  clusters:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                      required:
+                      - name
+                      type: object
+                    type: array
+                type: object
+              template:
+                properties:
+                  metadata:
+                    type: object
+                  spec:
+                    description: UserSpec defines the desired state of User
+                    properties:
+                      description:
+                        description: Description of the user.
+                        type: string
+                      displayName:
+                        type: string
+                      email:
+                        description: Unique email address(https://www.ietf.org/rfc/rfc5322.txt).
+                        type: string
+                      groups:
+                        items:
+                          type: string
+                        type: array
+                      lang:
+                        description: The preferred written or spoken language for the user.
+                        type: string
+                      password:
+                        description: password will be encrypted by mutating admission webhook
+                        type: string
+                    required:
+                    - email
+                    type: object
+                  status:
+                    description: UserStatus defines the observed state of User
+                    properties:
+                      lastLoginTime:
+                        description: Last login attempt timestamp
+                        format: date-time
+                        type: string
+                      lastTransitionTime:
+                        format: date-time
+                        type: string
+                      reason:
+                        type: string
+                      state:
+                        description: The user status
+                        type: string
+                    type: object
+                required:
+                - spec
+                type: object
+            required:
+            - placement
+            - template
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_globalrolebindings.yaml

@@ -0,0 +1,83 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: globalrolebindings.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: GlobalRoleBinding
+    listKind: GlobalRoleBindingList
+    plural: globalrolebindings
+    singular: globalrolebinding
+  scope: Cluster
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: GlobalRoleBinding is the Schema for the globalrolebindings API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          roleRef:
+            description: RoleRef can only reference a GlobalRole. If the RoleRef cannot be resolved, the Authorizer must return an error.
+            properties:
+              apiGroup:
+                description: APIGroup is the group for the resource being referenced
+                type: string
+              kind:
+                description: Kind is the type of resource being referenced
+                type: string
+              name:
+                description: Name is the name of resource being referenced
+                type: string
+            required:
+            - apiGroup
+            - kind
+            - name
+            type: object
+          subjects:
+            description: Subjects holds references to the objects the role applies to.
+            items:
+              description: Subject contains a reference to the object or user identities a role binding applies to.  This can either hold a direct API object reference, or a value for non-objects such as user and group names.
+              properties:
+                apiGroup:
+                  description: APIGroup holds the API group of the referenced subject. Defaults to "" for ServiceAccount subjects. Defaults to "rbac.authorization.k8s.io" for User and Group subjects.
+                  type: string
+                kind:
+                  description: Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount". If the Authorizer does not recognized the kind value, the Authorizer should report an error.
+                  type: string
+                name:
+                  description: Name of the object being referenced.
+                  type: string
+                namespace:
+                  description: Namespace of the referenced object.  If the object kind is non-namespace, such as "User" or "Group", and this value is not empty the Authorizer should report an error.
+                  type: string
+              required:
+              - kind
+              - name
+              type: object
+            type: array
+        required:
+        - roleRef
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_globalroles.yaml

@@ -0,0 +1,75 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: globalroles.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: GlobalRole
+    listKind: GlobalRoleList
+    plural: globalroles
+    singular: globalrole
+  scope: Cluster
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          rules:
+            description: Rules holds all the PolicyRules for this GlobalRole
+            items:
+              description: PolicyRule holds information that describes a policy rule, but does not contain information about who the rule applies to or which namespace the rule applies to.
+              properties:
+                apiGroups:
+                  description: APIGroups is the name of the APIGroup that contains the resources.  If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed.
+                  items:
+                    type: string
+                  type: array
+                nonResourceURLs:
+                  description: NonResourceURLs is a set of partial urls that a user should have access to.  *s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"),  but not both.
+                  items:
+                    type: string
+                  type: array
+                resourceNames:
+                  description: ResourceNames is an optional white list of names that the rule applies to.  An empty set means that everything is allowed.
+                  items:
+                    type: string
+                  type: array
+                resources:
+                  description: Resources is a list of resources this rule applies to.  ResourceAll represents all resources.
+                  items:
+                    type: string
+                  type: array
+                verbs:
+                  description: Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule.  VerbAll represents all kinds.
+                  items:
+                    type: string
+                  type: array
+              required:
+              - verbs
+              type: object
+            type: array
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_groupbindings.yaml

@@ -0,0 +1,64 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: groupbindings.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - group
+    kind: GroupBinding
+    listKind: GroupBindingList
+    plural: groupbindings
+    singular: groupbinding
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .groupRef.name
+      name: Group
+      type: string
+    - jsonPath: .users
+      name: Users
+      type: string
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: GroupBinding is the Schema for the groupbindings API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          groupRef:
+            description: GroupRef defines the desired relation of GroupBinding
+            properties:
+              apiGroup:
+                type: string
+              kind:
+                type: string
+              name:
+                type: string
+            type: object
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          users:
+            items:
+              type: string
+            type: array
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_groups.yaml

@@ -0,0 +1,53 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: groups.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - group
+    kind: Group
+    listKind: GroupList
+    plural: groups
+    singular: group
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: Workspace
+      type: string
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: Group is the Schema for the groups API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: GroupSpec defines the desired state of Group
+            type: object
+          status:
+            description: GroupStatus defines the observed state of Group
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_loginrecords.yaml

@@ -0,0 +1,90 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: loginrecords.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: LoginRecord
+    listKind: LoginRecordList
+    plural: loginrecords
+    singular: loginrecord
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.type
+      name: Type
+      type: string
+    - jsonPath: .spec.provider
+      name: Provider
+      type: string
+    - jsonPath: .spec.sourceIP
+      name: From
+      type: string
+    - jsonPath: .spec.success
+      name: Success
+      type: string
+    - jsonPath: .spec.reason
+      name: Reason
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              provider:
+                description: Provider of authentication, Ldap/Github etc.
+                type: string
+              reason:
+                description: States failed login attempt reason
+                type: string
+              sourceIP:
+                description: Source IP of client
+                type: string
+              success:
+                description: Successful login attempt or not
+                type: boolean
+              type:
+                description: Which authentication method used, BasicAuth/OAuth
+                type: string
+              userAgent:
+                description: User agent of login attempt
+                type: string
+            required:
+            - provider
+            - reason
+            - sourceIP
+            - success
+            - type
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_rolebases.yaml

@@ -0,0 +1,45 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: rolebases.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: RoleBase
+    listKind: RoleBaseList
+    plural: rolebases
+    singular: rolebase
+  scope: Cluster
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          role:
+            type: object
+        required:
+        - role
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_users.yaml

@@ -0,0 +1,92 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: users.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: User
+    listKind: UserList
+    plural: users
+    singular: user
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.email
+      name: Email
+      type: string
+    - jsonPath: .status.state
+      name: Status
+      type: string
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: User is the Schema for the users API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: UserSpec defines the desired state of User
+            properties:
+              description:
+                description: Description of the user.
+                type: string
+              displayName:
+                type: string
+              email:
+                description: Unique email address(https://www.ietf.org/rfc/rfc5322.txt).
+                type: string
+              groups:
+                items:
+                  type: string
+                type: array
+              lang:
+                description: The preferred written or spoken language for the user.
+                type: string
+              password:
+                description: password will be encrypted by mutating admission webhook
+                type: string
+            required:
+            - email
+            type: object
+          status:
+            description: UserStatus defines the observed state of User
+            properties:
+              lastLoginTime:
+                description: Last login attempt timestamp
+                format: date-time
+                type: string
+              lastTransitionTime:
+                format: date-time
+                type: string
+              reason:
+                type: string
+              state:
+                description: The user status
+                type: string
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_workspacerolebindings.yaml

@@ -0,0 +1,88 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: workspacerolebindings.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: WorkspaceRoleBinding
+    listKind: WorkspaceRoleBindingList
+    plural: workspacerolebindings
+    singular: workspacerolebinding
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: Workspace
+      type: string
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: WorkspaceRoleBinding is the Schema for the workspacerolebindings API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          roleRef:
+            description: RoleRef can only reference a WorkspaceRole. If the RoleRef cannot be resolved, the Authorizer must return an error.
+            properties:
+              apiGroup:
+                description: APIGroup is the group for the resource being referenced
+                type: string
+              kind:
+                description: Kind is the type of resource being referenced
+                type: string
+              name:
+                description: Name is the name of resource being referenced
+                type: string
+            required:
+            - apiGroup
+            - kind
+            - name
+            type: object
+          subjects:
+            description: Subjects holds references to the objects the role applies to.
+            items:
+              description: Subject contains a reference to the object or user identities a role binding applies to.  This can either hold a direct API object reference, or a value for non-objects such as user and group names.
+              properties:
+                apiGroup:
+                  description: APIGroup holds the API group of the referenced subject. Defaults to "" for ServiceAccount subjects. Defaults to "rbac.authorization.k8s.io" for User and Group subjects.
+                  type: string
+                kind:
+                  description: Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount". If the Authorizer does not recognized the kind value, the Authorizer should report an error.
+                  type: string
+                name:
+                  description: Name of the object being referenced.
+                  type: string
+                namespace:
+                  description: Namespace of the referenced object.  If the object kind is non-namespace, such as "User" or "Group", and this value is not empty the Authorizer should report an error.
+                  type: string
+              required:
+              - kind
+              - name
+              type: object
+            type: array
+        required:
+        - roleRef
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/iam.kubesphere.io_workspaceroles.yaml

@@ -0,0 +1,83 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: workspaceroles.iam.kubesphere.io
+spec:
+  group: iam.kubesphere.io
+  names:
+    categories:
+    - iam
+    kind: WorkspaceRole
+    listKind: WorkspaceRoleList
+    plural: workspaceroles
+    singular: workspacerole
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .metadata.labels.kubesphere\.io/workspace
+      name: Workspace
+      type: string
+    - jsonPath: .metadata.annotations.kubesphere\.io/alias-name
+      name: Alias
+      type: string
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          rules:
+            description: Rules holds all the PolicyRules for this WorkspaceRole
+            items:
+              description: PolicyRule holds information that describes a policy rule, but does not contain information about who the rule applies to or which namespace the rule applies to.
+              properties:
+                apiGroups:
+                  description: APIGroups is the name of the APIGroup that contains the resources.  If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed.
+                  items:
+                    type: string
+                  type: array
+                nonResourceURLs:
+                  description: NonResourceURLs is a set of partial urls that a user should have access to.  *s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"),  but not both.
+                  items:
+                    type: string
+                  type: array
+                resourceNames:
+                  description: ResourceNames is an optional white list of names that the rule applies to.  An empty set means that everything is allowed.
+                  items:
+                    type: string
+                  type: array
+                resources:
+                  description: Resources is a list of resources this rule applies to.  ResourceAll represents all resources.
+                  items:
+                    type: string
+                  type: array
+                verbs:
+                  description: Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule.  VerbAll represents all kinds.
+                  items:
+                    type: string
+                  type: array
+              required:
+              - verbs
+              type: object
+            type: array
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/monitoring.kubesphere.io_clusterdashboards.yaml

@@ -0,0 +1,184 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.2.4
+  creationTimestamp: null
+  name: clusterdashboards.monitoring.kubesphere.io
+spec:
+  group: monitoring.kubesphere.io
+  names:
+    kind: ClusterDashboard
+    listKind: ClusterDashboardList
+    plural: clusterdashboards
+    singular: clusterdashboard
+  scope: Cluster
+  validation:
+    openAPIV3Schema:
+      description: ClusterDashboard is the Schema for the culsterdashboards API
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: DashboardSpec defines the desired state of Dashboard
+          properties:
+            datasource:
+              description: Dashboard datasource
+              type: string
+            description:
+              description: Dashboard description
+              type: string
+            panels:
+              description: Collection of panels. Panel is one of [Row](row.md), [Singlestat](#singlestat.md)
+                or [Graph](graph.md)
+              items:
+                description: Supported panel type
+                properties:
+                  bars:
+                    description: Display as a bar chart
+                    type: boolean
+                  colors:
+                    description: Set series color
+                    items:
+                      type: string
+                    type: array
+                  decimals:
+                    description: Limit the decimal numbers
+                    format: int64
+                    type: integer
+                  description:
+                    description: Panel description
+                    type: string
+                  format:
+                    description: Display unit
+                    type: string
+                  id:
+                    description: Panel ID
+                    format: int64
+                    type: integer
+                  lines:
+                    description: Display as a line chart
+                    type: boolean
+                  stack:
+                    description: Display as a stacked chart
+                    type: boolean
+                  targets:
+                    allOf:
+                    - items:
+                        description: Query editor options
+                        properties:
+                          expr:
+                            description: Input for fetching metrics.
+                            type: string
+                          legendFormat:
+                            description: Legend format for outputs. You can make a
+                              dynamic legend with templating variables.
+                            type: string
+                          refId:
+                            description: Reference ID
+                            format: int64
+                            type: integer
+                          step:
+                            description: Set series time interval
+                            type: string
+                        type: object
+                    - items:
+                        description: Query editor options
+                        properties:
+                          expr:
+                            description: Input for fetching metrics.
+                            type: string
+                          legendFormat:
+                            description: Legend format for outputs. You can make a
+                              dynamic legend with templating variables.
+                            type: string
+                          refId:
+                            description: Reference ID
+                            format: int64
+                            type: integer
+                          step:
+                            description: Set series time interval
+                            type: string
+                        type: object
+                    description: A collection of queries
+                    type: array
+                  title:
+                    description: Name of the row panel
+                    type: string
+                  type:
+                    description: Must be `row`
+                    type: string
+                  yaxes:
+                    description: Y-axis options
+                    items:
+                      properties:
+                        decimals:
+                          description: Limit the decimal numbers
+                          format: int64
+                          type: integer
+                        format:
+                          description: Display unit
+                          type: string
+                      type: object
+                    type: array
+                required:
+                - type
+                type: object
+              type: array
+            templating:
+              description: Templating variables
+              items:
+                description: Templating defines a variable, which can be used as a
+                  placeholder in query
+                properties:
+                  name:
+                    description: Variable name
+                    type: string
+                  query:
+                    description: Set variable values to be the return result of the
+                      query
+                    type: string
+                type: object
+              type: array
+            time:
+              description: Time range for display
+              properties:
+                from:
+                  description: Start time in the format of `^now([+-][0-9]+[smhdwMy])?$`,
+                    eg. `now-1M`. It denotes the end time is set to the last month
+                    since now.
+                  type: string
+                to:
+                  description: End time in the format of `^now([+-][0-9]+[smhdwMy])?$`,
+                    eg. `now-1M`. It denotes the start time is set to the last month
+                    since now.
+                  type: string
+              type: object
+            title:
+              description: Dashboard title
+              type: string
+          type: object
+      type: object
+  version: v1alpha1
+  versions:
+  - name: v1alpha1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/monitoring.kubesphere.io_dashboards.yaml

@@ -0,0 +1,184 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.2.4
+  creationTimestamp: null
+  name: dashboards.monitoring.kubesphere.io
+spec:
+  group: monitoring.kubesphere.io
+  names:
+    kind: Dashboard
+    listKind: DashboardList
+    plural: dashboards
+    singular: dashboard
+  scope: Namespaced
+  validation:
+    openAPIV3Schema:
+      description: Dashboard is the Schema for the dashboards API
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        metadata:
+          type: object
+        spec:
+          description: DashboardSpec defines the desired state of Dashboard
+          properties:
+            datasource:
+              description: Dashboard datasource
+              type: string
+            description:
+              description: Dashboard description
+              type: string
+            panels:
+              description: Collection of panels. Panel is one of [Row](row.md), [Singlestat](#singlestat.md)
+                or [Graph](graph.md)
+              items:
+                description: Supported panel type
+                properties:
+                  bars:
+                    description: Display as a bar chart
+                    type: boolean
+                  colors:
+                    description: Set series color
+                    items:
+                      type: string
+                    type: array
+                  decimals:
+                    description: Limit the decimal numbers
+                    format: int64
+                    type: integer
+                  description:
+                    description: Panel description
+                    type: string
+                  format:
+                    description: Display unit
+                    type: string
+                  id:
+                    description: Panel ID
+                    format: int64
+                    type: integer
+                  lines:
+                    description: Display as a line chart
+                    type: boolean
+                  stack:
+                    description: Display as a stacked chart
+                    type: boolean
+                  targets:
+                    allOf:
+                    - items:
+                        description: Query editor options
+                        properties:
+                          expr:
+                            description: Input for fetching metrics.
+                            type: string
+                          legendFormat:
+                            description: Legend format for outputs. You can make a
+                              dynamic legend with templating variables.
+                            type: string
+                          refId:
+                            description: Reference ID
+                            format: int64
+                            type: integer
+                          step:
+                            description: Set series time interval
+                            type: string
+                        type: object
+                    - items:
+                        description: Query editor options
+                        properties:
+                          expr:
+                            description: Input for fetching metrics.
+                            type: string
+                          legendFormat:
+                            description: Legend format for outputs. You can make a
+                              dynamic legend with templating variables.
+                            type: string
+                          refId:
+                            description: Reference ID
+                            format: int64
+                            type: integer
+                          step:
+                            description: Set series time interval
+                            type: string
+                        type: object
+                    description: A collection of queries
+                    type: array
+                  title:
+                    description: Name of the row panel
+                    type: string
+                  type:
+                    description: Must be `row`
+                    type: string
+                  yaxes:
+                    description: Y-axis options
+                    items:
+                      properties:
+                        decimals:
+                          description: Limit the decimal numbers
+                          format: int64
+                          type: integer
+                        format:
+                          description: Display unit
+                          type: string
+                      type: object
+                    type: array
+                required:
+                - type
+                type: object
+              type: array
+            templating:
+              description: Templating variables
+              items:
+                description: Templating defines a variable, which can be used as a
+                  placeholder in query
+                properties:
+                  name:
+                    description: Variable name
+                    type: string
+                  query:
+                    description: Set variable values to be the return result of the
+                      query
+                    type: string
+                type: object
+              type: array
+            time:
+              description: Time range for display
+              properties:
+                from:
+                  description: Start time in the format of `^now([+-][0-9]+[smhdwMy])?$`,
+                    eg. `now-1M`. It denotes the end time is set to the last month
+                    since now.
+                  type: string
+                to:
+                  description: End time in the format of `^now([+-][0-9]+[smhdwMy])?$`,
+                    eg. `now-1M`. It denotes the start time is set to the last month
+                    since now.
+                  type: string
+              type: object
+            title:
+              description: Dashboard title
+              type: string
+          type: object
+      type: object
+  version: v1alpha1
+  versions:
+  - name: v1alpha1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/network.kubesphere.io_ipamblocks.yaml

@@ -0,0 +1,76 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: ipamblocks.network.kubesphere.io
+spec:
+  group: network.kubesphere.io
+  names:
+    kind: IPAMBlock
+    listKind: IPAMBlockList
+    plural: ipamblocks
+    singular: ipamblock
+  scope: Cluster
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Specification of the IPAMBlock.
+            properties:
+              allocations:
+                items:
+                  type: integer
+                type: array
+              attributes:
+                items:
+                  properties:
+                    handle_id:
+                      type: string
+                    secondary:
+                      additionalProperties:
+                        type: string
+                      type: object
+                  type: object
+                type: array
+              cidr:
+                type: string
+              deleted:
+                type: boolean
+              id:
+                format: int32
+                type: integer
+              unallocated:
+                items:
+                  type: integer
+                type: array
+            required:
+            - allocations
+            - attributes
+            - cidr
+            - deleted
+            - id
+            - unallocated
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/network.kubesphere.io_ipamhandles.yaml

@@ -0,0 +1,55 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: ipamhandles.network.kubesphere.io
+spec:
+  group: network.kubesphere.io
+  names:
+    kind: IPAMHandle
+    listKind: IPAMHandleList
+    plural: ipamhandles
+    singular: ipamhandle
+  scope: Cluster
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Specification of the IPAMHandle.
+            properties:
+              block:
+                additionalProperties:
+                  type: integer
+                type: object
+              deleted:
+                type: boolean
+              handleID:
+                type: string
+            required:
+            - block
+            - deleted
+            - handleID
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/network.kubesphere.io_ippools.yaml

@@ -0,0 +1,130 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: ippools.network.kubesphere.io
+spec:
+  group: network.kubesphere.io
+  names:
+    kind: IPPool
+    listKind: IPPoolList
+    plural: ippools
+    singular: ippool
+  scope: Cluster
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              blockSize:
+                description: The block size to use for IP address assignments from this pool. Defaults to 26 for IPv4 and 112 for IPv6.
+                type: integer
+              cidr:
+                description: The pool CIDR.
+                type: string
+              disabled:
+                description: When disabled is true, IPAM will not assign addresses from this pool.
+                type: boolean
+              dns:
+                description: DNS contains values interesting for DNS resolvers
+                properties:
+                  domain:
+                    type: string
+                  nameservers:
+                    items:
+                      type: string
+                    type: array
+                  options:
+                    items:
+                      type: string
+                    type: array
+                  search:
+                    items:
+                      type: string
+                    type: array
+                type: object
+              gateway:
+                type: string
+              rangeEnd:
+                description: The last ip, inclusive
+                type: string
+              rangeStart:
+                description: The first ip, inclusive
+                type: string
+              routes:
+                items:
+                  properties:
+                    dst:
+                      type: string
+                    gateway:
+                      type: string
+                  type: object
+                type: array
+              type:
+                type: string
+              vlanConfig:
+                properties:
+                  master:
+                    type: string
+                  vlanId:
+                    format: int32
+                    type: integer
+                required:
+                - master
+                - vlanId
+                type: object
+            required:
+            - cidr
+            - type
+            type: object
+          status:
+            properties:
+              allocations:
+                type: integer
+              capacity:
+                type: integer
+              reserved:
+                type: integer
+              synced:
+                type: boolean
+              unallocated:
+                type: integer
+              workspaces:
+                additionalProperties:
+                  properties:
+                    allocations:
+                      type: integer
+                  required:
+                  - allocations
+                  type: object
+                type: object
+            required:
+            - allocations
+            - capacity
+            - unallocated
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/network.kubesphere.io_namespacenetworkpolicies.yaml

@@ -0,0 +1,179 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: namespacenetworkpolicies.network.kubesphere.io
+spec:
+  group: network.kubesphere.io
+  names:
+    categories:
+    - networking
+    kind: NamespaceNetworkPolicy
+    listKind: NamespaceNetworkPolicyList
+    plural: namespacenetworkpolicies
+    shortNames:
+    - nsnp
+    singular: namespacenetworkpolicy
+  scope: Namespaced
+  preserveUnknownFields: false
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: NamespaceNetworkPolicy is the Schema for the namespacenetworkpolicies API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: NamespaceNetworkPolicySpec provides the specification of a NamespaceNetworkPolicy
+            properties:
+              egress:
+                description: List of egress rules to be applied to the selected pods. Outgoing traffic is allowed if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic matches at least one egress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy limits all outgoing traffic (and serves solely to ensure that the pods it selects are isolated by default). This field is beta-level in 1.8
+                items:
+                  description: NetworkPolicyEgressRule describes a particular set of traffic that is allowed out of pods matched by a NetworkPolicySpec's podSelector. The traffic must match both ports and to. This type is beta-level in 1.8
+                  properties:
+                    ports:
+                      description: List of destination ports for outgoing traffic. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list.
+                      items:
+                        description: NetworkPolicyPort describes a port to allow traffic on
+                        properties:
+                          port:
+                            anyOf:
+                            - type: integer
+                            - type: string
+                            description: The port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers.
+                            x-kubernetes-int-or-string: true
+                          protocol:
+                            default: TCP
+                            description: The protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP.
+                            type: string
+                        type: object
+                      type: array
+                    to:
+                      description: List of destinations for outgoing traffic of pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all destinations (traffic not restricted by destination). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the to list.
+                      items:
+                        description: NetworkPolicyPeer describes a peer to allow traffic from. Only certain combinations of fields are allowed
+                        properties:
+                          ipBlock:
+                            description: IPBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be.
+                            properties:
+                              cidr:
+                                description: CIDR is a string representing the IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64"
+                                type: string
+                              except:
+                                description: Except is a slice of CIDRs that should not be included within an IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64" Except values will be rejected if they are outside the CIDR range
+                                items:
+                                  type: string
+                                type: array
+                            required:
+                            - cidr
+                            type: object
+                          namespace:
+                            properties:
+                              name:
+                                type: string
+                            required:
+                            - name
+                            type: object
+                          service:
+                            properties:
+                              name:
+                                type: string
+                              namespace:
+                                type: string
+                            required:
+                            - name
+                            - namespace
+                            type: object
+                        type: object
+                      type: array
+                  type: object
+                type: array
+              ingress:
+                description: List of ingress rules to be applied to the selected pods. Traffic is allowed to a pod if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic source is the pod's local node, OR if the traffic matches at least one ingress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy does not allow any traffic (and serves solely to ensure that the pods it selects are isolated by default)
+                items:
+                  description: NetworkPolicyIngressRule describes a particular set of traffic that is allowed to the pods matched by a NetworkPolicySpec's podSelector. The traffic must match both ports and from.
+                  properties:
+                    from:
+                      description: List of sources which should be able to access the pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all sources (traffic not restricted by source). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the from list.
+                      items:
+                        description: NetworkPolicyPeer describes a peer to allow traffic from. Only certain combinations of fields are allowed
+                        properties:
+                          ipBlock:
+                            description: IPBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be.
+                            properties:
+                              cidr:
+                                description: CIDR is a string representing the IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64"
+                                type: string
+                              except:
+                                description: Except is a slice of CIDRs that should not be included within an IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64" Except values will be rejected if they are outside the CIDR range
+                                items:
+                                  type: string
+                                type: array
+                            required:
+                            - cidr
+                            type: object
+                          namespace:
+                            properties:
+                              name:
+                                type: string
+                            required:
+                            - name
+                            type: object
+                          service:
+                            properties:
+                              name:
+                                type: string
+                              namespace:
+                                type: string
+                            required:
+                            - name
+                            - namespace
+                            type: object
+                        type: object
+                      type: array
+                    ports:
+                      description: List of ports which should be made accessible on the pods selected for this rule. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list.
+                      items:
+                        description: NetworkPolicyPort describes a port to allow traffic on
+                        properties:
+                          port:
+                            anyOf:
+                            - type: integer
+                            - type: string
+                            description: The port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers.
+                            x-kubernetes-int-or-string: true
+                          protocol:
+                            default: TCP
+                            description: The protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP.
+                            type: string
+                        type: object
+                      type: array
+                  type: object
+                type: array
+              policyTypes:
+                description: List of rule types that the NetworkPolicy relates to. Valid options are "Ingress", "Egress", or "Ingress,Egress". If this field is not specified, it will default based on the existence of Ingress or Egress rules; policies that contain an Egress section are assumed to affect Egress, and all policies (whether or not they contain an Ingress section) are assumed to affect Ingress. If you want to write an egress-only policy, you must explicitly specify policyTypes [ "Egress" ]. Likewise, if you want to write a policy that specifies that no egress is allowed, you must specify a policyTypes value that include "Egress" (since such a policy would not include an Egress section and would otherwise default to just [ "Ingress" ]). This field is beta-level in 1.8
+                items:
+                  description: Policy Type string describes the NetworkPolicy type This type is beta-level in 1.8
+                  type: string
+                type: array
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/notification.kubesphere.io_configs.yaml

@@ -0,0 +1,283 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: configs.notification.kubesphere.io
+spec:
+  group: notification.kubesphere.io
+  names:
+    categories:
+    - notification-manager
+    kind: Config
+    listKind: ConfigList
+    plural: configs
+    shortNames:
+    - nc
+    singular: config
+  scope: Cluster
+  versions:
+  - name: v2beta1
+    schema:
+      openAPIV3Schema:
+        description: DingTalkConfig is the Schema for the dingtalkconfigs API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: ConfigSpec defines the desired state of Config
+            properties:
+              dingtalk:
+                properties:
+                  conversation:
+                    description: Only needed when send alerts to the conversation.
+                    properties:
+                      appkey:
+                        description: The key of the application with which to send messages.
+                        properties:
+                          key:
+                            description: The key of the secret to select from.  Must be a valid secret key.
+                            type: string
+                          name:
+                            description: Name of the secret.
+                            type: string
+                          namespace:
+                            description: The namespace of the secret, default to the pod's namespace.
+                            type: string
+                        required:
+                        - key
+                        type: object
+                      appsecret:
+                        description: The key in the secret to be used. Must be a valid secret key.
+                        properties:
+                          key:
+                            description: The key of the secret to select from.  Must be a valid secret key.
+                            type: string
+                          name:
+                            description: Name of the secret.
+                            type: string
+                          namespace:
+                            description: The namespace of the secret, default to the pod's namespace.
+                            type: string
+                        required:
+                        - key
+                        type: object
+                    type: object
+                  labels:
+                    additionalProperties:
+                      type: string
+                    type: object
+                type: object
+              email:
+                properties:
+                  authIdentify:
+                    description: The identity for PLAIN authentication.
+                    type: string
+                  authPassword:
+                    description: The secret contains the SMTP password for LOGIN and PLAIN authentications.
+                    properties:
+                      key:
+                        description: The key of the secret to select from.  Must be a valid secret key.
+                        type: string
+                      name:
+                        description: Name of the secret.
+                        type: string
+                      namespace:
+                        description: The namespace of the secret, default to the pod's namespace.
+                        type: string
+                    required:
+                    - key
+                    type: object
+                  authSecret:
+                    description: The secret contains the SMTP secret for CRAM-MD5 authentication.
+                    properties:
+                      key:
+                        description: The key of the secret to select from.  Must be a valid secret key.
+                        type: string
+                      name:
+                        description: Name of the secret.
+                        type: string
+                      namespace:
+                        description: The namespace of the secret, default to the pod's namespace.
+                        type: string
+                    required:
+                    - key
+                    type: object
+                  authUsername:
+                    description: The username for CRAM-MD5, LOGIN and PLAIN authentications.
+                    type: string
+                  from:
+                    description: The sender address.
+                    type: string
+                  hello:
+                    description: The hostname to use when identifying to the SMTP server.
+                    type: string
+                  labels:
+                    additionalProperties:
+                      type: string
+                    type: object
+                  requireTLS:
+                    description: The default SMTP TLS requirement.
+                    type: boolean
+                  smartHost:
+                    description: The address of the SMTP server.
+                    properties:
+                      host:
+                        type: string
+                      port:
+                        type: integer
+                    required:
+                    - host
+                    - port
+                    type: object
+                  tls:
+                    description: TLSConfig configures the options for TLS connections.
+                    properties:
+                      clientCertificate:
+                        description: The certificate of the client.
+                        properties:
+                          cert:
+                            description: The client cert file for the targets.
+                            properties:
+                              key:
+                                description: The key of the secret to select from.  Must be a valid secret key.
+                                type: string
+                              name:
+                                description: Name of the secret.
+                                type: string
+                              namespace:
+                                description: The namespace of the secret, default to the pod's namespace.
+                                type: string
+                            required:
+                            - key
+                            type: object
+                          key:
+                            description: The client key file for the targets.
+                            properties:
+                              key:
+                                description: The key of the secret to select from.  Must be a valid secret key.
+                                type: string
+                              name:
+                                description: Name of the secret.
+                                type: string
+                              namespace:
+                                description: The namespace of the secret, default to the pod's namespace.
+                                type: string
+                            required:
+                            - key
+                            type: object
+                        type: object
+                      insecureSkipVerify:
+                        description: Disable target certificate validation.
+                        type: boolean
+                      rootCA:
+                        description: RootCA defines the root certificate authorities that clients use when verifying server certificates.
+                        properties:
+                          key:
+                            description: The key of the secret to select from.  Must be a valid secret key.
+                            type: string
+                          name:
+                            description: Name of the secret.
+                            type: string
+                          namespace:
+                            description: The namespace of the secret, default to the pod's namespace.
+                            type: string
+                        required:
+                        - key
+                        type: object
+                      serverName:
+                        description: Used to verify the hostname for the targets.
+                        type: string
+                    type: object
+                required:
+                - from
+                - smartHost
+                type: object
+              slack:
+                properties:
+                  labels:
+                    additionalProperties:
+                      type: string
+                    type: object
+                  slackTokenSecret:
+                    description: The token of user or bot.
+                    properties:
+                      key:
+                        description: The key of the secret to select from.  Must be a valid secret key.
+                        type: string
+                      name:
+                        description: Name of the secret.
+                        type: string
+                      namespace:
+                        description: The namespace of the secret, default to the pod's namespace.
+                        type: string
+                    required:
+                    - key
+                    type: object
+                type: object
+              webhook:
+                properties:
+                  labels:
+                    additionalProperties:
+                      type: string
+                    type: object
+                type: object
+              wechat:
+                properties:
+                  labels:
+                    additionalProperties:
+                      type: string
+                    type: object
+                  wechatApiAgentId:
+                    description: The id of the application which sending message.
+                    type: string
+                  wechatApiCorpId:
+                    description: The corp id for authentication.
+                    type: string
+                  wechatApiSecret:
+                    description: The API key to use when talking to the WeChat API.
+                    properties:
+                      key:
+                        description: The key of the secret to select from.  Must be a valid secret key.
+                        type: string
+                      name:
+                        description: Name of the secret.
+                        type: string
+                      namespace:
+                        description: The namespace of the secret, default to the pod's namespace.
+                        type: string
+                    required:
+                    - key
+                    type: object
+                  wechatApiUrl:
+                    description: The WeChat API URL.
+                    type: string
+                required:
+                - wechatApiAgentId
+                - wechatApiCorpId
+                - wechatApiSecret
+                type: object
+            type: object
+          status:
+            description: ConfigStatus defines the observed state of Config
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/notification.kubesphere.io_receivers.yaml

@@ -0,0 +1,590 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: receivers.notification.kubesphere.io
+spec:
+  group: notification.kubesphere.io
+  names:
+    categories:
+    - notification-manager
+    kind: Receiver
+    listKind: ReceiverList
+    plural: receivers
+    shortNames:
+    - nr
+    singular: receiver
+  scope: Cluster
+  versions:
+  - name: v2beta1
+    schema:
+      openAPIV3Schema:
+        description: Receiver is the Schema for the receivers API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: ReceiverSpec defines the desired state of Receiver
+            properties:
+              dingtalk:
+                properties:
+                  alertSelector:
+                    description: Selector to filter alerts.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  chatbot:
+                    description: Be careful, a ChatBot only can send 20 message per minute.
+                    properties:
+                      keywords:
+                        description: Custom keywords of ChatBot
+                        items:
+                          type: string
+                        type: array
+                      secret:
+                        description: Secret of ChatBot, you can get it after enabled Additional Signature of ChatBot.
+                        properties:
+                          key:
+                            description: The key of the secret to select from.  Must be a valid secret key.
+                            type: string
+                          name:
+                            description: Name of the secret.
+                            type: string
+                          namespace:
+                            description: The namespace of the secret, default to the pod's namespace.
+                            type: string
+                        required:
+                        - key
+                        type: object
+                      webhook:
+                        description: The webhook of ChatBot which the message will send to.
+                        properties:
+                          key:
+                            description: The key of the secret to select from.  Must be a valid secret key.
+                            type: string
+                          name:
+                            description: Name of the secret.
+                            type: string
+                          namespace:
+                            description: The namespace of the secret, default to the pod's namespace.
+                            type: string
+                        required:
+                        - key
+                        type: object
+                    required:
+                    - webhook
+                    type: object
+                  conversation:
+                    description: The conversation which message will send to.
+                    properties:
+                      chatids:
+                        items:
+                          type: string
+                        type: array
+                    required:
+                    - chatids
+                    type: object
+                  dingtalkConfigSelector:
+                    description: DingTalkConfig to be selected for this receiver
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  enabled:
+                    description: whether the receiver is enabled
+                    type: boolean
+                type: object
+              email:
+                properties:
+                  alertSelector:
+                    description: Selector to filter alerts.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  emailConfigSelector:
+                    description: EmailConfig to be selected for this receiver
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  enabled:
+                    description: whether the receiver is enabled
+                    type: boolean
+                  to:
+                    description: Receivers' email addresses
+                    items:
+                      type: string
+                    type: array
+                required:
+                - to
+                type: object
+              slack:
+                properties:
+                  alertSelector:
+                    description: Selector to filter alerts.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  channels:
+                    description: The channel or user to send notifications to.
+                    items:
+                      type: string
+                    type: array
+                  enabled:
+                    description: whether the receiver is enabled
+                    type: boolean
+                  slackConfigSelector:
+                    description: SlackConfig to be selected for this receiver
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                required:
+                - channels
+                type: object
+              webhook:
+                properties:
+                  alertSelector:
+                    description: Selector to filter alerts.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  enabled:
+                    description: whether the receiver is enabled
+                    type: boolean
+                  httpConfig:
+                    description: HTTPClientConfig configures an HTTP client.
+                    properties:
+                      basicAuth:
+                        description: The HTTP basic authentication credentials for the targets.
+                        properties:
+                          password:
+                            description: SecretKeySelector selects a key of a Secret.
+                            properties:
+                              key:
+                                description: The key of the secret to select from.  Must be a valid secret key.
+                                type: string
+                              name:
+                                description: Name of the secret.
+                                type: string
+                              namespace:
+                                description: The namespace of the secret, default to the pod's namespace.
+                                type: string
+                            required:
+                            - key
+                            type: object
+                          username:
+                            type: string
+                        required:
+                        - username
+                        type: object
+                      bearerToken:
+                        description: The bearer token for the targets.
+                        properties:
+                          key:
+                            description: The key of the secret to select from.  Must be a valid secret key.
+                            type: string
+                          name:
+                            description: Name of the secret.
+                            type: string
+                          namespace:
+                            description: The namespace of the secret, default to the pod's namespace.
+                            type: string
+                        required:
+                        - key
+                        type: object
+                      proxyUrl:
+                        description: HTTP proxy server to use to connect to the targets.
+                        type: string
+                      tlsConfig:
+                        description: TLSConfig to use to connect to the targets.
+                        properties:
+                          clientCertificate:
+                            description: The certificate of the client.
+                            properties:
+                              cert:
+                                description: The client cert file for the targets.
+                                properties:
+                                  key:
+                                    description: The key of the secret to select from.  Must be a valid secret key.
+                                    type: string
+                                  name:
+                                    description: Name of the secret.
+                                    type: string
+                                  namespace:
+                                    description: The namespace of the secret, default to the pod's namespace.
+                                    type: string
+                                required:
+                                - key
+                                type: object
+                              key:
+                                description: The client key file for the targets.
+                                properties:
+                                  key:
+                                    description: The key of the secret to select from.  Must be a valid secret key.
+                                    type: string
+                                  name:
+                                    description: Name of the secret.
+                                    type: string
+                                  namespace:
+                                    description: The namespace of the secret, default to the pod's namespace.
+                                    type: string
+                                required:
+                                - key
+                                type: object
+                            type: object
+                          insecureSkipVerify:
+                            description: Disable target certificate validation.
+                            type: boolean
+                          rootCA:
+                            description: RootCA defines the root certificate authorities that clients use when verifying server certificates.
+                            properties:
+                              key:
+                                description: The key of the secret to select from.  Must be a valid secret key.
+                                type: string
+                              name:
+                                description: Name of the secret.
+                                type: string
+                              namespace:
+                                description: The namespace of the secret, default to the pod's namespace.
+                                type: string
+                            required:
+                            - key
+                            type: object
+                          serverName:
+                            description: Used to verify the hostname for the targets.
+                            type: string
+                        type: object
+                    type: object
+                  service:
+                    description: "`service` is a reference to the service for this webhook. Either `service` or `url` must be specified. \n If the webhook is running within the cluster, then you should use `service`."
+                    properties:
+                      name:
+                        description: '`name` is the name of the service. Required'
+                        type: string
+                      namespace:
+                        description: '`namespace` is the namespace of the service. Required'
+                        type: string
+                      path:
+                        description: '`path` is an optional URL path which will be sent in any request to this service.'
+                        type: string
+                      port:
+                        description: If specified, the port on the service that hosting webhook. Default to 443 for backward compatibility. `port` should be a valid port number (1-65535, inclusive).
+                        format: int32
+                        type: integer
+                      scheme:
+                        description: Http scheme, default is http.
+                        type: string
+                    required:
+                    - name
+                    - namespace
+                    type: object
+                  url:
+                    description: "`url` gives the location of the webhook, in standard URL form (`scheme://host:port/path`). Exactly one of `url` or `service` must be specified. \n The `host` should not refer to a service running in the cluster; use the `service` field instead. The host might be resolved via external DNS in some api servers (e.g., `kube-apiserver` cannot resolve in-cluster DNS as that would be a layering violation). `host` may also be an IP address. \n Please note that using `localhost` or `127.0.0.1` as a `host` is risky unless you take great care to run this webhook on all hosts which run an apiserver which might need to make calls to this webhook. Such installs are likely to be non-portable, i.e., not easy to turn up in a new cluster. \n A path is optional, and if present may be any string permissible in a URL. You may use the path to pass an arbitrary string to the webhook, for example, a cluster identifier. \n Attempting to use a user or basic auth e.g. \"user:password@\" is not allowed. Fragments (\"#...\") and query parameters (\"?...\") are not allowed, either."
+                    type: string
+                  webhookConfigSelector:
+                    description: WebhookConfig to be selected for this receiver
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                type: object
+              wechat:
+                properties:
+                  alertSelector:
+                    description: Selector to filter alerts.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  enabled:
+                    description: whether the receiver is enabled
+                    type: boolean
+                  toParty:
+                    items:
+                      type: string
+                    type: array
+                  toTag:
+                    items:
+                      type: string
+                    type: array
+                  toUser:
+                    items:
+                      type: string
+                    type: array
+                  wechatConfigSelector:
+                    description: WechatConfig to be selected for this receiver
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                type: object
+            type: object
+          status:
+            description: ReceiverStatus defines the observed state of Receiver
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/quota.kubesphere.io_resourcequotas.yaml

@@ -0,0 +1,161 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: resourcequotas.quota.kubesphere.io
+spec:
+  group: quota.kubesphere.io
+  names:
+    categories:
+    - quota
+    kind: ResourceQuota
+    listKind: ResourceQuotaList
+    plural: resourcequotas
+    singular: resourcequota
+  scope: Cluster
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: WorkspaceResourceQuota sets aggregate quota restrictions enforced per workspace
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Spec defines the desired quota
+            properties:
+              quota:
+                description: Quota defines the desired quota
+                properties:
+                  hard:
+                    additionalProperties:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                    description: 'hard is the set of desired hard limits for each named resource. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/'
+                    type: object
+                  scopeSelector:
+                    description: scopeSelector is also a collection of filters like scopes that must match each object tracked by a quota but expressed using ScopeSelectorOperator in combination with possible values. For a resource to match, both scopes AND scopeSelector (if specified in spec), must be matched.
+                    properties:
+                      matchExpressions:
+                        description: A list of scope selector requirements by scope of the resources.
+                        items:
+                          description: A scoped-resource selector requirement is a selector that contains values, a scope name, and an operator that relates the scope name and values.
+                          properties:
+                            operator:
+                              description: Represents a scope's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist.
+                              type: string
+                            scopeName:
+                              description: The name of the scope that the selector applies to.
+                              type: string
+                            values:
+                              description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - operator
+                          - scopeName
+                          type: object
+                        type: array
+                    type: object
+                  scopes:
+                    description: A collection of filters that must match each object tracked by a quota. If not specified, the quota matches all objects.
+                    items:
+                      description: A ResourceQuotaScope defines a filter that must match each object tracked by a quota
+                      type: string
+                    type: array
+                type: object
+              selector:
+                additionalProperties:
+                  type: string
+                description: LabelSelector is used to select projects by label.
+                type: object
+            required:
+            - quota
+            - selector
+            type: object
+          status:
+            description: Status defines the actual enforced quota and its current usage
+            properties:
+              namespaces:
+                description: Namespaces slices the usage by project.
+                items:
+                  description: ResourceQuotaStatusByNamespace gives status for a particular project
+                  properties:
+                    hard:
+                      additionalProperties:
+                        anyOf:
+                        - type: integer
+                        - type: string
+                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                        x-kubernetes-int-or-string: true
+                      description: 'Hard is the set of enforced hard limits for each named resource. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/'
+                      type: object
+                    namespace:
+                      description: Namespace the project this status applies to
+                      type: string
+                    used:
+                      additionalProperties:
+                        anyOf:
+                        - type: integer
+                        - type: string
+                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                        x-kubernetes-int-or-string: true
+                      description: Used is the current observed total usage of the resource in the namespace.
+                      type: object
+                  required:
+                  - namespace
+                  type: object
+                type: array
+              total:
+                description: Total defines the actual enforced quota and its current usage across all projects
+                properties:
+                  hard:
+                    additionalProperties:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                    description: 'Hard is the set of enforced hard limits for each named resource. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/'
+                    type: object
+                  used:
+                    additionalProperties:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                    description: Used is the current observed total usage of the resource in the namespace.
+                    type: object
+                type: object
+            required:
+            - namespaces
+            - total
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/servicemesh.kubesphere.io_strategies.yaml

@@ -0,0 +1,671 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: strategies.servicemesh.kubesphere.io
+spec:
+  group: servicemesh.kubesphere.io
+  names:
+    kind: Strategy
+    listKind: StrategyList
+    plural: strategies
+    singular: strategy
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - description: type of strategy
+      jsonPath: .spec.type
+      name: Type
+      type: string
+    - description: destination hosts
+      jsonPath: .spec.template.spec.hosts
+      name: Hosts
+      type: string
+    - description: 'CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC. Populated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata'
+      jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: Strategy is the Schema for the strategies API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: StrategySpec defines the desired state of Strategy
+            properties:
+              governor:
+                description: Governor version, the version takes control of all incoming traffic label version value
+                type: string
+              principal:
+                description: Principal version, the one as reference version label version value
+                type: string
+              selector:
+                description: Label selector for virtual services.
+                properties:
+                  matchExpressions:
+                    description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                    items:
+                      description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                      properties:
+                        key:
+                          description: key is the label key that the selector applies to.
+                          type: string
+                        operator:
+                          description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                          type: string
+                        values:
+                          description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                          items:
+                            type: string
+                          type: array
+                      required:
+                      - key
+                      - operator
+                      type: object
+                    type: array
+                  matchLabels:
+                    additionalProperties:
+                      type: string
+                    description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                    type: object
+                type: object
+              strategyPolicy:
+                description: strategy policy, how the strategy will be applied by the strategy controller
+                type: string
+              template:
+                description: Template describes the virtual service that will be created.
+                properties:
+                  metadata:
+                    description: Metadata of the virtual services created from this template
+                    type: object
+                  spec:
+                    description: Spec indicates the behavior of a virtual service.
+                    properties:
+                      export_to:
+                        description: "A list of namespaces to which this virtual service is exported. Exporting a virtual service allows it to be used by sidecars and gateways defined in other namespaces. This feature provides a mechanism for service owners and mesh administrators to control the visibility of virtual services across namespace boundaries. \n If no namespaces are specified then the virtual service is exported to all namespaces by default. \n The value \".\" is reserved and defines an export to the same namespace that the virtual service is declared in. Similarly the value \"*\" is reserved and defines an export to all namespaces. \n NOTE: in the current release, the `exportTo` value is restricted to \".\" or \"*\" (i.e., the current namespace or all namespaces)."
+                        items:
+                          type: string
+                        type: array
+                      gateways:
+                        description: The names of gateways and sidecars that should apply these routes. Gateways in other namespaces may be referred to by `<gateway namespace>/<gateway name>`; specifying a gateway with no namespace qualifier is the same as specifying the VirtualService's namespace. A single VirtualService is used for sidecars inside the mesh as well as for one or more gateways. The selection condition imposed by this field can be overridden using the source field in the match conditions of protocol-specific routes. The reserved word `mesh` is used to imply all the sidecars in the mesh. When this field is omitted, the default gateway (`mesh`) will be used, which would apply the rule to all sidecars in the mesh. If a list of gateway names is provided, the rules will apply only to the gateways. To apply the rules to both gateways and sidecars, specify `mesh` as one of the gateway names.
+                        items:
+                          type: string
+                        type: array
+                      hosts:
+                        description: "The destination hosts to which traffic is being sent. Could be a DNS name with wildcard prefix or an IP address.  Depending on the platform, short-names can also be used instead of a FQDN (i.e. has no dots in the name). In such a scenario, the FQDN of the host would be derived based on the underlying platform. \n A single VirtualService can be used to describe all the traffic properties of the corresponding hosts, including those for multiple HTTP and TCP ports. Alternatively, the traffic properties of a host can be defined using more than one VirtualService, with certain caveats. Refer to the [Operations Guide](https://istio.io/docs/ops/best-practices/traffic-management/#split-virtual-services) for details. \n *Note for Kubernetes users*: When short names are used (e.g. \"reviews\" instead of \"reviews.default.svc.cluster.local\"), Istio will interpret the short name based on the namespace of the rule, not the service. A rule in the \"default\" namespace containing a host \"reviews\" will be interpreted as \"reviews.default.svc.cluster.local\", irrespective of the actual namespace associated with the reviews service. _To avoid potential misconfigurations, it is recommended to always use fully qualified domain names over short names._ \n The hosts field applies to both HTTP and TCP services. Service inside the mesh, i.e., those found in the service registry, must always be referred to using their alphanumeric names. IP addresses are allowed only for services defined via the Gateway. \n *Note*: It must be empty for a delegate VirtualService."
+                        items:
+                          type: string
+                        type: array
+                      http:
+                        description: An ordered list of route rules for HTTP traffic. HTTP routes will be applied to platform service ports named 'http-*'/'http2-*'/'grpc-*', gateway ports with protocol HTTP/HTTP2/GRPC/ TLS-terminated-HTTPS and service entry ports using HTTP/HTTP2/GRPC protocols.  The first rule matching an incoming request is used.
+                        items:
+                          description: Describes match conditions and actions for routing HTTP/1.1, HTTP2, and gRPC traffic. See VirtualService for usage examples.
+                          properties:
+                            cors_policy:
+                              description: Cross-Origin Resource Sharing policy (CORS). Refer to [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) for further details about cross origin resource sharing.
+                              properties:
+                                allow_credentials:
+                                  description: Indicates whether the caller is allowed to send the actual request (not the preflight) using credentials. Translates to `Access-Control-Allow-Credentials` header.
+                                  properties:
+                                    value:
+                                      description: The bool value.
+                                      type: boolean
+                                  type: object
+                                allow_headers:
+                                  description: List of HTTP headers that can be used when requesting the resource. Serialized to Access-Control-Allow-Headers header.
+                                  items:
+                                    type: string
+                                  type: array
+                                allow_methods:
+                                  description: List of HTTP methods allowed to access the resource. The content will be serialized into the Access-Control-Allow-Methods header.
+                                  items:
+                                    type: string
+                                  type: array
+                                allow_origin:
+                                  description: The list of origins that are allowed to perform CORS requests. The content will be serialized into the Access-Control-Allow-Origin header. Wildcard * will allow all origins. $hide_from_docs
+                                  items:
+                                    type: string
+                                  type: array
+                                allow_origins:
+                                  description: String patterns that match allowed origins. An origin is allowed if any of the string matchers match. If a match is found, then the outgoing Access-Control-Allow-Origin would be set to the origin as provided by the client.
+                                  items:
+                                    description: Describes how to match a given string in HTTP headers. Match is case-sensitive.
+                                    type: object
+                                  type: array
+                                expose_headers:
+                                  description: A white list of HTTP headers that the browsers are allowed to access. Serialized into Access-Control-Expose-Headers header.
+                                  items:
+                                    type: string
+                                  type: array
+                                max_age:
+                                  description: Specifies how long the results of a preflight request can be cached. Translates to the `Access-Control-Max-Age` header.
+                                  properties:
+                                    nanos:
+                                      description: Signed fractions of a second at nanosecond resolution of the span of time. Durations less than one second are represented with a 0 `seconds` field and a positive or negative `nanos` field. For durations of one second or more, a non-zero value for the `nanos` field must be of the same sign as the `seconds` field. Must be from -999,999,999 to +999,999,999 inclusive.
+                                      format: int32
+                                      type: integer
+                                    seconds:
+                                      description: 'Signed seconds of the span of time. Must be from -315,576,000,000 to +315,576,000,000 inclusive. Note: these bounds are computed from: 60 sec/min * 60 min/hr * 24 hr/day * 365.25 days/year * 10000 years'
+                                      format: int64
+                                      type: integer
+                                  type: object
+                              type: object
+                            delegate:
+                              description: 'Delegate is used to specify the particular VirtualService which can be used to define delegate HTTPRoute. It can be set only when `Route` and `Redirect` are empty, and the route rules of the delegate VirtualService will be merged with that in the current one. **NOTE**:    1. Only one level delegation is supported.    2. The delegate''s HTTPMatchRequest must be a strict subset of the root''s,       otherwise there is a conflict and the HTTPRoute will not take effect.'
+                              properties:
+                                name:
+                                  description: Name specifies the name of the delegate VirtualService.
+                                  type: string
+                                namespace:
+                                  description: Namespace specifies the namespace where the delegate VirtualService resides. By default, it is same to the root's.
+                                  type: string
+                              type: object
+                            fault:
+                              description: Fault injection policy to apply on HTTP traffic at the client side. Note that timeouts or retries will not be enabled when faults are enabled on the client side.
+                              properties:
+                                abort:
+                                  description: Abort Http request attempts and return error codes back to downstream service, giving the impression that the upstream service is faulty.
+                                  properties:
+                                    percentage:
+                                      description: Percentage of requests to be aborted with the error code provided.
+                                      properties:
+                                        value:
+                                          type: number
+                                      type: object
+                                  type: object
+                                delay:
+                                  description: Delay requests before forwarding, emulating various failures such as network issues, overloaded upstream service, etc.
+                                  properties:
+                                    percent:
+                                      description: Percentage of requests on which the delay will be injected (0-100). Use of integer `percent` value is deprecated. Use the double `percentage` field instead.
+                                      format: int32
+                                      type: integer
+                                    percentage:
+                                      description: Percentage of requests on which the delay will be injected.
+                                      properties:
+                                        value:
+                                          type: number
+                                      type: object
+                                  type: object
+                              type: object
+                            headers:
+                              description: Header manipulation rules
+                              properties:
+                                request:
+                                  description: Header manipulation rules to apply before forwarding a request to the destination service
+                                  properties:
+                                    add:
+                                      additionalProperties:
+                                        type: string
+                                      description: Append the given values to the headers specified by keys (will create a comma-separated list of values)
+                                      type: object
+                                    remove:
+                                      description: Remove a the specified headers
+                                      items:
+                                        type: string
+                                      type: array
+                                    set:
+                                      additionalProperties:
+                                        type: string
+                                      description: Overwrite the headers specified by key with the given values
+                                      type: object
+                                  type: object
+                                response:
+                                  description: Header manipulation rules to apply before returning a response to the caller
+                                  properties:
+                                    add:
+                                      additionalProperties:
+                                        type: string
+                                      description: Append the given values to the headers specified by keys (will create a comma-separated list of values)
+                                      type: object
+                                    remove:
+                                      description: Remove a the specified headers
+                                      items:
+                                        type: string
+                                      type: array
+                                    set:
+                                      additionalProperties:
+                                        type: string
+                                      description: Overwrite the headers specified by key with the given values
+                                      type: object
+                                  type: object
+                              type: object
+                            match:
+                              description: Match conditions to be satisfied for the rule to be activated. All conditions inside a single match block have AND semantics, while the list of match blocks have OR semantics. The rule is matched if any one of the match blocks succeed.
+                              items:
+                                description: "HttpMatchRequest specifies a set of criterion to be met in order for the rule to be applied to the HTTP request. For example, the following restricts the rule to match only requests where the URL path starts with /ratings/v2/ and the request contains a custom `end-user` header with value `jason`. \n {{<tabset category-name=\"example\">}} {{<tab name=\"v1alpha3\" category-value=\"v1alpha3\">}} ```yaml apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata:   name: ratings-route spec:   hosts:   - ratings.prod.svc.cluster.local   http:   - match:     - headers:         end-user:           exact: jason       uri:         prefix: \"/ratings/v2/\"       ignoreUriCase: true     route:     - destination:         host: ratings.prod.svc.cluster.local ``` {{</tab>}} \n {{<tab name=\"v1beta1\" category-value=\"v1beta1\">}} ```yaml apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata:   name: ratings-route spec:   hosts:   - ratings.prod.svc.cluster.local   http:   - match:     - headers:         end-user:           exact: jason       uri:         prefix: \"/ratings/v2/\"       ignoreUriCase: true     route:     - destination:         host: ratings.prod.svc.cluster.local ``` {{</tab>}} {{</tabset>}} \n HTTPMatchRequest CANNOT be empty. **Note:** No regex string match can be set when delegate VirtualService is specified."
+                                properties:
+                                  authority:
+                                    description: "HTTP Authority values are case-sensitive and formatted as follows: \n - `exact: \"value\"` for exact string match \n - `prefix: \"value\"` for prefix-based match \n - `regex: \"value\"` for ECMAscript style regex-based match"
+                                    type: object
+                                  gateways:
+                                    description: Names of gateways where the rule should be applied. Gateway names in the top-level `gateways` field of the VirtualService (if any) are overridden. The gateway match is independent of sourceLabels.
+                                    items:
+                                      type: string
+                                    type: array
+                                  headers:
+                                    additionalProperties:
+                                      description: Describes how to match a given string in HTTP headers. Match is case-sensitive.
+                                      type: object
+                                    description: "The header keys must be lowercase and use hyphen as the separator, e.g. _x-request-id_. \n Header values are case-sensitive and formatted as follows: \n - `exact: \"value\"` for exact string match \n - `prefix: \"value\"` for prefix-based match \n - `regex: \"value\"` for ECMAscript style regex-based match \n If the value is empty and only the name of header is specfied, presence of the header is checked. **Note:** The keys `uri`, `scheme`, `method`, and `authority` will be ignored."
+                                    type: object
+                                  ignore_uri_case:
+                                    description: "Flag to specify whether the URI matching should be case-insensitive. \n **Note:** The case will be ignored only in the case of `exact` and `prefix` URI matches."
+                                    type: boolean
+                                  method:
+                                    description: "HTTP Method values are case-sensitive and formatted as follows: \n - `exact: \"value\"` for exact string match \n - `prefix: \"value\"` for prefix-based match \n - `regex: \"value\"` for ECMAscript style regex-based match"
+                                    type: object
+                                  name:
+                                    description: The name assigned to a match. The match's name will be concatenated with the parent route's name and will be logged in the access logs for requests matching this route.
+                                    type: string
+                                  port:
+                                    description: Specifies the ports on the host that is being addressed. Many services only expose a single port or label ports with the protocols they support, in these cases it is not required to explicitly select the port.
+                                    format: int32
+                                    type: integer
+                                  query_params:
+                                    additionalProperties:
+                                      description: Describes how to match a given string in HTTP headers. Match is case-sensitive.
+                                      type: object
+                                    description: "Query parameters for matching. \n Ex: - For a query parameter like \"?key=true\", the map key would be \"key\" and   the string match could be defined as `exact: \"true\"`. - For a query parameter like \"?key\", the map key would be \"key\" and the   string match could be defined as `exact: \"\"`. - For a query parameter like \"?key=123\", the map key would be \"key\" and the   string match could be defined as `regex: \"\\d+$\"`. Note that this   configuration will only match values like \"123\" but not \"a123\" or \"123a\". \n **Note:** `prefix` matching is currently not supported."
+                                    type: object
+                                  scheme:
+                                    description: "URI Scheme values are case-sensitive and formatted as follows: \n - `exact: \"value\"` for exact string match \n - `prefix: \"value\"` for prefix-based match \n - `regex: \"value\"` for ECMAscript style regex-based match"
+                                    type: object
+                                  source_labels:
+                                    additionalProperties:
+                                      type: string
+                                    description: One or more labels that constrain the applicability of a rule to workloads with the given labels. If the VirtualService has a list of gateways specified in the top-level `gateways` field, it must include the reserved gateway `mesh` for this field to be applicable.
+                                    type: object
+                                  source_namespace:
+                                    description: Source namespace constraining the applicability of a rule to workloads in that namespace. If the VirtualService has a list of gateways specified in the top-level `gateways` field, it must include the reserved gateway `mesh` for this field to be applicable.
+                                    type: string
+                                  uri:
+                                    description: "URI to match values are case-sensitive and formatted as follows: \n - `exact: \"value\"` for exact string match \n - `prefix: \"value\"` for prefix-based match \n - `regex: \"value\"` for ECMAscript style regex-based match \n **Note:** Case-insensitive matching could be enabled via the `ignore_uri_case` flag."
+                                    type: object
+                                  without_headers:
+                                    additionalProperties:
+                                      description: Describes how to match a given string in HTTP headers. Match is case-sensitive.
+                                      type: object
+                                    description: withoutHeader has the same syntax with the header, but has opposite meaning. If a header is matched with a matching rule among withoutHeader, the traffic becomes not matched one.
+                                    type: object
+                                type: object
+                              type: array
+                            mirror:
+                              description: Mirror HTTP traffic to a another destination in addition to forwarding the requests to the intended destination. Mirrored traffic is on a best effort basis where the sidecar/gateway will not wait for the mirrored cluster to respond before returning the response from the original destination.  Statistics will be generated for the mirrored destination.
+                              properties:
+                                host:
+                                  description: "The name of a service from the service registry. Service names are looked up from the platform's service registry (e.g., Kubernetes services, Consul services, etc.) and from the hosts declared by [ServiceEntry](https://istio.io/docs/reference/config/networking/service-entry/#ServiceEntry). Traffic forwarded to destinations that are not found in either of the two, will be dropped. \n *Note for Kubernetes users*: When short names are used (e.g. \"reviews\" instead of \"reviews.default.svc.cluster.local\"), Istio will interpret the short name based on the namespace of the rule, not the service. A rule in the \"default\" namespace containing a host \"reviews will be interpreted as \"reviews.default.svc.cluster.local\", irrespective of the actual namespace associated with the reviews service. To avoid potential misconfiguration, it is recommended to always use fully qualified domain names over short names."
+                                  type: string
+                                port:
+                                  description: Specifies the port on the host that is being addressed. If a service exposes only a single port it is not required to explicitly select the port.
+                                  properties:
+                                    number:
+                                      description: Valid port number
+                                      format: int32
+                                      type: integer
+                                  type: object
+                                subset:
+                                  description: The name of a subset within the service. Applicable only to services within the mesh. The subset must be defined in a corresponding DestinationRule.
+                                  type: string
+                              type: object
+                            mirror_percent:
+                              description: Percentage of the traffic to be mirrored by the `mirror` field. Use of integer `mirror_percent` value is deprecated. Use the double `mirror_percentage` field instead
+                              properties:
+                                value:
+                                  description: The uint32 value.
+                                  format: int32
+                                  type: integer
+                              type: object
+                            mirror_percentage:
+                              description: Percentage of the traffic to be mirrored by the `mirror` field. If this field is absent, all the traffic (100%) will be mirrored. Max value is 100.
+                              properties:
+                                value:
+                                  type: number
+                              type: object
+                            name:
+                              description: The name assigned to the route for debugging purposes. The route's name will be concatenated with the match's name and will be logged in the access logs for requests matching this route/match.
+                              type: string
+                            redirect:
+                              description: A HTTP rule can either redirect or forward (default) traffic. If traffic passthrough option is specified in the rule, route/redirect will be ignored. The redirect primitive can be used to send a HTTP 301 redirect to a different URI or Authority.
+                              properties:
+                                authority:
+                                  description: On a redirect, overwrite the Authority/Host portion of the URL with this value.
+                                  type: string
+                                redirect_code:
+                                  description: On a redirect, Specifies the HTTP status code to use in the redirect response. The default response code is MOVED_PERMANENTLY (301).
+                                  format: int32
+                                  type: integer
+                                uri:
+                                  description: On a redirect, overwrite the Path portion of the URL with this value. Note that the entire path will be replaced, irrespective of the request URI being matched as an exact path or prefix.
+                                  type: string
+                              type: object
+                            retries:
+                              description: Retry policy for HTTP requests.
+                              properties:
+                                attempts:
+                                  description: Number of retries for a given request. The interval between retries will be determined automatically (25ms+). Actual number of retries attempted depends on the request `timeout` of the [HTTP route](https://istio.io/docs/reference/config/networking/virtual-service/#HTTPRoute).
+                                  format: int32
+                                  type: integer
+                                per_try_timeout:
+                                  description: 'Timeout per retry attempt for a given request. format: 1h/1m/1s/1ms. MUST BE >=1ms.'
+                                  properties:
+                                    nanos:
+                                      description: Signed fractions of a second at nanosecond resolution of the span of time. Durations less than one second are represented with a 0 `seconds` field and a positive or negative `nanos` field. For durations of one second or more, a non-zero value for the `nanos` field must be of the same sign as the `seconds` field. Must be from -999,999,999 to +999,999,999 inclusive.
+                                      format: int32
+                                      type: integer
+                                    seconds:
+                                      description: 'Signed seconds of the span of time. Must be from -315,576,000,000 to +315,576,000,000 inclusive. Note: these bounds are computed from: 60 sec/min * 60 min/hr * 24 hr/day * 365.25 days/year * 10000 years'
+                                      format: int64
+                                      type: integer
+                                  type: object
+                                retry_on:
+                                  description: Specifies the conditions under which retry takes place. One or more policies can be specified using a ‘,’ delimited list. See the [retry policies](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/router_filter#x-envoy-retry-on) and [gRPC retry policies](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/router_filter#x-envoy-retry-grpc-on) for more details.
+                                  type: string
+                                retry_remote_localities:
+                                  description: Flag to specify whether the retries should retry to other localities. See the [retry plugin configuration](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/http_connection_management#retry-plugin-configuration) for more details.
+                                  properties:
+                                    value:
+                                      description: The bool value.
+                                      type: boolean
+                                  type: object
+                              type: object
+                            rewrite:
+                              description: Rewrite HTTP URIs and Authority headers. Rewrite cannot be used with Redirect primitive. Rewrite will be performed before forwarding.
+                              properties:
+                                authority:
+                                  description: rewrite the Authority/Host header with this value.
+                                  type: string
+                                uri:
+                                  description: rewrite the path (or the prefix) portion of the URI with this value. If the original URI was matched based on prefix, the value provided in this field will replace the corresponding matched prefix.
+                                  type: string
+                              type: object
+                            route:
+                              description: A HTTP rule can either redirect or forward (default) traffic. The forwarding target can be one of several versions of a service (see glossary in beginning of document). Weights associated with the service version determine the proportion of traffic it receives.
+                              items:
+                                description: "Each routing rule is associated with one or more service versions (see glossary in beginning of document). Weights associated with the version determine the proportion of traffic it receives. For example, the following rule will route 25% of traffic for the \"reviews\" service to instances with the \"v2\" tag and the remaining traffic (i.e., 75%) to \"v1\". \n {{<tabset category-name=\"example\">}} {{<tab name=\"v1alpha3\" category-value=\"v1alpha3\">}} ```yaml apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata:   name: reviews-route spec:   hosts:   - reviews.prod.svc.cluster.local   http:   - route:     - destination:         host: reviews.prod.svc.cluster.local         subset: v2       weight: 25     - destination:         host: reviews.prod.svc.cluster.local         subset: v1       weight: 75 ``` {{</tab>}} \n {{<tab name=\"v1beta1\" category-value=\"v1beta1\">}} ```yaml apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata:   name: reviews-route spec:   hosts:   - reviews.prod.svc.cluster.local   http:   - route:     - destination:         host: reviews.prod.svc.cluster.local         subset: v2       weight: 25     - destination:         host: reviews.prod.svc.cluster.local         subset: v1       weight: 75 ``` {{</tab>}} {{</tabset>}} \n And the associated DestinationRule \n {{<tabset category-name=\"example\">}} {{<tab name=\"v1alpha3\" category-value=\"v1alpha3\">}} ```yaml apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata:   name: reviews-destination spec:   host: reviews.prod.svc.cluster.local   subsets:   - name: v1     labels:       version: v1   - name: v2     labels:       version: v2 ``` {{</tab>}} \n {{<tab name=\"v1beta1\" category-value=\"v1beta1\">}} ```yaml apiVersion: networking.istio.io/v1beta1 kind: DestinationRule metadata:   name: reviews-destination spec:   host: reviews.prod.svc.cluster.local   subsets:   - name: v1     labels:       version: v1   - name: v2     labels:       version: v2 ``` {{</tab>}} {{</tabset>}} \n Traffic can also be split across two entirely different services without having to define new subsets. For example, the following rule forwards 25% of traffic to reviews.com to dev.reviews.com \n {{<tabset category-name=\"example\">}} {{<tab name=\"v1alpha3\" category-value=\"v1alpha3\">}} ```yaml apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata:   name: reviews-route-two-domains spec:   hosts:   - reviews.com   http:   - route:     - destination:         host: dev.reviews.com       weight: 25     - destination:         host: reviews.com       weight: 75 ``` {{</tab>}} \n {{<tab name=\"v1beta1\" category-value=\"v1beta1\">}} ```yaml apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata:   name: reviews-route-two-domains spec:   hosts:   - reviews.com   http:   - route:     - destination:         host: dev.reviews.com       weight: 25     - destination:         host: reviews.com       weight: 75 ``` {{</tab>}} {{</tabset>}}"
+                                properties:
+                                  destination:
+                                    description: Destination uniquely identifies the instances of a service to which the request/connection should be forwarded to.
+                                    properties:
+                                      host:
+                                        description: "The name of a service from the service registry. Service names are looked up from the platform's service registry (e.g., Kubernetes services, Consul services, etc.) and from the hosts declared by [ServiceEntry](https://istio.io/docs/reference/config/networking/service-entry/#ServiceEntry). Traffic forwarded to destinations that are not found in either of the two, will be dropped. \n *Note for Kubernetes users*: When short names are used (e.g. \"reviews\" instead of \"reviews.default.svc.cluster.local\"), Istio will interpret the short name based on the namespace of the rule, not the service. A rule in the \"default\" namespace containing a host \"reviews will be interpreted as \"reviews.default.svc.cluster.local\", irrespective of the actual namespace associated with the reviews service. To avoid potential misconfiguration, it is recommended to always use fully qualified domain names over short names."
+                                        type: string
+                                      port:
+                                        description: Specifies the port on the host that is being addressed. If a service exposes only a single port it is not required to explicitly select the port.
+                                        properties:
+                                          number:
+                                            description: Valid port number
+                                            format: int32
+                                            type: integer
+                                        type: object
+                                      subset:
+                                        description: The name of a subset within the service. Applicable only to services within the mesh. The subset must be defined in a corresponding DestinationRule.
+                                        type: string
+                                    type: object
+                                  headers:
+                                    description: Header manipulation rules
+                                    properties:
+                                      request:
+                                        description: Header manipulation rules to apply before forwarding a request to the destination service
+                                        properties:
+                                          add:
+                                            additionalProperties:
+                                              type: string
+                                            description: Append the given values to the headers specified by keys (will create a comma-separated list of values)
+                                            type: object
+                                          remove:
+                                            description: Remove a the specified headers
+                                            items:
+                                              type: string
+                                            type: array
+                                          set:
+                                            additionalProperties:
+                                              type: string
+                                            description: Overwrite the headers specified by key with the given values
+                                            type: object
+                                        type: object
+                                      response:
+                                        description: Header manipulation rules to apply before returning a response to the caller
+                                        properties:
+                                          add:
+                                            additionalProperties:
+                                              type: string
+                                            description: Append the given values to the headers specified by keys (will create a comma-separated list of values)
+                                            type: object
+                                          remove:
+                                            description: Remove a the specified headers
+                                            items:
+                                              type: string
+                                            type: array
+                                          set:
+                                            additionalProperties:
+                                              type: string
+                                            description: Overwrite the headers specified by key with the given values
+                                            type: object
+                                        type: object
+                                    type: object
+                                  weight:
+                                    description: The proportion of traffic to be forwarded to the service version. (0-100). Sum of weights across destinations SHOULD BE == 100. If there is only one destination in a rule, the weight value is assumed to be 100.
+                                    format: int32
+                                    type: integer
+                                type: object
+                              type: array
+                            timeout:
+                              description: Timeout for HTTP requests.
+                              properties:
+                                nanos:
+                                  description: Signed fractions of a second at nanosecond resolution of the span of time. Durations less than one second are represented with a 0 `seconds` field and a positive or negative `nanos` field. For durations of one second or more, a non-zero value for the `nanos` field must be of the same sign as the `seconds` field. Must be from -999,999,999 to +999,999,999 inclusive.
+                                  format: int32
+                                  type: integer
+                                seconds:
+                                  description: 'Signed seconds of the span of time. Must be from -315,576,000,000 to +315,576,000,000 inclusive. Note: these bounds are computed from: 60 sec/min * 60 min/hr * 24 hr/day * 365.25 days/year * 10000 years'
+                                  format: int64
+                                  type: integer
+                              type: object
+                          type: object
+                        type: array
+                      tcp:
+                        description: An ordered list of route rules for opaque TCP traffic. TCP routes will be applied to any port that is not a HTTP or TLS port. The first rule matching an incoming request is used.
+                        items:
+                          description: "Describes match conditions and actions for routing TCP traffic. The following routing rule forwards traffic arriving at port 27017 for mongo.prod.svc.cluster.local to another Mongo server on port 5555. \n {{<tabset category-name=\"example\">}} {{<tab name=\"v1alpha3\" category-value=\"v1alpha3\">}} ```yaml apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata:   name: bookinfo-Mongo spec:   hosts:   - mongo.prod.svc.cluster.local   tcp:   - match:     - port: 27017     route:     - destination:         host: mongo.backup.svc.cluster.local         port:           number: 5555 ``` {{</tab>}} \n {{<tab name=\"v1beta1\" category-value=\"v1beta1\">}} ```yaml apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata:   name: bookinfo-Mongo spec:   hosts:   - mongo.prod.svc.cluster.local   tcp:   - match:     - port: 27017     route:     - destination:         host: mongo.backup.svc.cluster.local         port:           number: 5555 ``` {{</tab>}} {{</tabset>}}"
+                          properties:
+                            match:
+                              description: Match conditions to be satisfied for the rule to be activated. All conditions inside a single match block have AND semantics, while the list of match blocks have OR semantics. The rule is matched if any one of the match blocks succeed.
+                              items:
+                                description: L4 connection match attributes. Note that L4 connection matching support is incomplete.
+                                properties:
+                                  destination_subnets:
+                                    description: IPv4 or IPv6 ip addresses of destination with optional subnet.  E.g., a.b.c.d/xx form or just a.b.c.d.
+                                    items:
+                                      type: string
+                                    type: array
+                                  gateways:
+                                    description: Names of gateways where the rule should be applied. Gateway names in the top-level `gateways` field of the VirtualService (if any) are overridden. The gateway match is independent of sourceLabels.
+                                    items:
+                                      type: string
+                                    type: array
+                                  port:
+                                    description: Specifies the port on the host that is being addressed. Many services only expose a single port or label ports with the protocols they support, in these cases it is not required to explicitly select the port.
+                                    format: int32
+                                    type: integer
+                                  source_labels:
+                                    additionalProperties:
+                                      type: string
+                                    description: One or more labels that constrain the applicability of a rule to workloads with the given labels. If the VirtualService has a list of gateways specified in the top-level `gateways` field, it should include the reserved gateway `mesh` in order for this field to be applicable.
+                                    type: object
+                                  source_namespace:
+                                    description: Source namespace constraining the applicability of a rule to workloads in that namespace. If the VirtualService has a list of gateways specified in the top-level `gateways` field, it must include the reserved gateway `mesh` for this field to be applicable.
+                                    type: string
+                                  source_subnet:
+                                    description: IPv4 or IPv6 ip address of source with optional subnet. E.g., a.b.c.d/xx form or just a.b.c.d $hide_from_docs
+                                    type: string
+                                type: object
+                              type: array
+                            route:
+                              description: The destination to which the connection should be forwarded to.
+                              items:
+                                description: L4 routing rule weighted destination.
+                                properties:
+                                  destination:
+                                    description: Destination uniquely identifies the instances of a service to which the request/connection should be forwarded to.
+                                    properties:
+                                      host:
+                                        description: "The name of a service from the service registry. Service names are looked up from the platform's service registry (e.g., Kubernetes services, Consul services, etc.) and from the hosts declared by [ServiceEntry](https://istio.io/docs/reference/config/networking/service-entry/#ServiceEntry). Traffic forwarded to destinations that are not found in either of the two, will be dropped. \n *Note for Kubernetes users*: When short names are used (e.g. \"reviews\" instead of \"reviews.default.svc.cluster.local\"), Istio will interpret the short name based on the namespace of the rule, not the service. A rule in the \"default\" namespace containing a host \"reviews will be interpreted as \"reviews.default.svc.cluster.local\", irrespective of the actual namespace associated with the reviews service. To avoid potential misconfiguration, it is recommended to always use fully qualified domain names over short names."
+                                        type: string
+                                      port:
+                                        description: Specifies the port on the host that is being addressed. If a service exposes only a single port it is not required to explicitly select the port.
+                                        properties:
+                                          number:
+                                            description: Valid port number
+                                            format: int32
+                                            type: integer
+                                        type: object
+                                      subset:
+                                        description: The name of a subset within the service. Applicable only to services within the mesh. The subset must be defined in a corresponding DestinationRule.
+                                        type: string
+                                    type: object
+                                  weight:
+                                    description: The proportion of traffic to be forwarded to the service version. If there is only one destination in a rule, all traffic will be routed to it irrespective of the weight.
+                                    format: int32
+                                    type: integer
+                                type: object
+                              type: array
+                          type: object
+                        type: array
+                      tls:
+                        description: 'An ordered list of route rule for non-terminated TLS & HTTPS traffic. Routing is typically performed using the SNI value presented by the ClientHello message. TLS routes will be applied to platform service ports named ''https-*'', ''tls-*'', unterminated gateway ports using HTTPS/TLS protocols (i.e. with "passthrough" TLS mode) and service entry ports using HTTPS/TLS protocols.  The first rule matching an incoming request is used.  NOTE: Traffic ''https-*'' or ''tls-*'' ports without associated virtual service will be treated as opaque TCP traffic.'
+                        items:
+                          description: "Describes match conditions and actions for routing unterminated TLS traffic (TLS/HTTPS) The following routing rule forwards unterminated TLS traffic arriving at port 443 of gateway called \"mygateway\" to internal services in the mesh based on the SNI value. \n {{<tabset category-name=\"example\">}} {{<tab name=\"v1alpha3\" category-value=\"v1alpha3\">}} ```yaml apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata:   name: bookinfo-sni spec:   hosts:   - \"*.bookinfo.com\"   gateways:   - mygateway   tls:   - match:     - port: 443       sniHosts:       - login.bookinfo.com     route:     - destination:         host: login.prod.svc.cluster.local   - match:     - port: 443       sniHosts:       - reviews.bookinfo.com     route:     - destination:         host: reviews.prod.svc.cluster.local ``` {{</tab>}} \n {{<tab name=\"v1beta1\" category-value=\"v1beta1\">}} ```yaml apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata:   name: bookinfo-sni spec:   hosts:   - \"*.bookinfo.com\"   gateways:   - mygateway   tls:   - match:     - port: 443       sniHosts:       - login.bookinfo.com     route:     - destination:         host: login.prod.svc.cluster.local   - match:     - port: 443       sniHosts:       - reviews.bookinfo.com     route:     - destination:         host: reviews.prod.svc.cluster.local ``` {{</tab>}} {{</tabset>}}"
+                          properties:
+                            match:
+                              description: Match conditions to be satisfied for the rule to be activated. All conditions inside a single match block have AND semantics, while the list of match blocks have OR semantics. The rule is matched if any one of the match blocks succeed.
+                              items:
+                                description: TLS connection match attributes.
+                                properties:
+                                  destination_subnets:
+                                    description: IPv4 or IPv6 ip addresses of destination with optional subnet.  E.g., a.b.c.d/xx form or just a.b.c.d.
+                                    items:
+                                      type: string
+                                    type: array
+                                  gateways:
+                                    description: Names of gateways where the rule should be applied. Gateway names in the top-level `gateways` field of the VirtualService (if any) are overridden. The gateway match is independent of sourceLabels.
+                                    items:
+                                      type: string
+                                    type: array
+                                  port:
+                                    description: Specifies the port on the host that is being addressed. Many services only expose a single port or label ports with the protocols they support, in these cases it is not required to explicitly select the port.
+                                    format: int32
+                                    type: integer
+                                  sni_hosts:
+                                    description: SNI (server name indicator) to match on. Wildcard prefixes can be used in the SNI value, e.g., *.com will match foo.example.com as well as example.com. An SNI value must be a subset (i.e., fall within the domain) of the corresponding virtual serivce's hosts.
+                                    items:
+                                      type: string
+                                    type: array
+                                  source_labels:
+                                    additionalProperties:
+                                      type: string
+                                    description: One or more labels that constrain the applicability of a rule to workloads with the given labels. If the VirtualService has a list of gateways specified in the top-level `gateways` field, it should include the reserved gateway `mesh` in order for this field to be applicable.
+                                    type: object
+                                  source_namespace:
+                                    description: Source namespace constraining the applicability of a rule to workloads in that namespace. If the VirtualService has a list of gateways specified in the top-level `gateways` field, it must include the reserved gateway `mesh` for this field to be applicable.
+                                    type: string
+                                type: object
+                              type: array
+                            route:
+                              description: The destination to which the connection should be forwarded to.
+                              items:
+                                description: L4 routing rule weighted destination.
+                                properties:
+                                  destination:
+                                    description: Destination uniquely identifies the instances of a service to which the request/connection should be forwarded to.
+                                    properties:
+                                      host:
+                                        description: "The name of a service from the service registry. Service names are looked up from the platform's service registry (e.g., Kubernetes services, Consul services, etc.) and from the hosts declared by [ServiceEntry](https://istio.io/docs/reference/config/networking/service-entry/#ServiceEntry). Traffic forwarded to destinations that are not found in either of the two, will be dropped. \n *Note for Kubernetes users*: When short names are used (e.g. \"reviews\" instead of \"reviews.default.svc.cluster.local\"), Istio will interpret the short name based on the namespace of the rule, not the service. A rule in the \"default\" namespace containing a host \"reviews will be interpreted as \"reviews.default.svc.cluster.local\", irrespective of the actual namespace associated with the reviews service. To avoid potential misconfiguration, it is recommended to always use fully qualified domain names over short names."
+                                        type: string
+                                      port:
+                                        description: Specifies the port on the host that is being addressed. If a service exposes only a single port it is not required to explicitly select the port.
+                                        properties:
+                                          number:
+                                            description: Valid port number
+                                            format: int32
+                                            type: integer
+                                        type: object
+                                      subset:
+                                        description: The name of a subset within the service. Applicable only to services within the mesh. The subset must be defined in a corresponding DestinationRule.
+                                        type: string
+                                    type: object
+                                  weight:
+                                    description: The proportion of traffic to be forwarded to the service version. If there is only one destination in a rule, all traffic will be routed to it irrespective of the weight.
+                                    format: int32
+                                    type: integer
+                                type: object
+                              type: array
+                          type: object
+                        type: array
+                    type: object
+                type: object
+              type:
+                description: Strategy type
+                type: string
+            type: object
+          status:
+            description: StrategyStatus defines the observed state of Strategy
+            properties:
+              completionTime:
+                description: Represents time when the strategy was completed. It is represented in RFC3339 form and is in UTC.
+                format: date-time
+                type: string
+              conditions:
+                description: The latest available observations of an object's current state.
+                items:
+                  description: StrategyCondition describes current state of a strategy.
+                  properties:
+                    lastProbeTime:
+                      description: Last time the condition was checked.
+                      format: date-time
+                      type: string
+                    lastTransitionTime:
+                      description: Last time the condition transit from one status to another
+                      format: date-time
+                      type: string
+                    message:
+                      description: Human readable message indicating details about last transition.
+                      type: string
+                    reason:
+                      description: reason for the condition's last transition
+                      type: string
+                    status:
+                      description: Status of the condition, one of True, False, Unknown
+                      type: string
+                    type:
+                      description: Type of strategy condition, Complete or Failed.
+                      type: string
+                  type: object
+                type: array
+              startTime:
+                description: Represents time when the strategy was acknowledged by the controller. It is represented in RFC3339 form and is in UTC.
+                format: date-time
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/servicemesh_v1alpha2_servicepolicy.yaml

@@ -1,863 +0,0 @@
-apiVersion: apiextensions.k8s.io/v1beta1
-kind: CustomResourceDefinition
-metadata:
-  creationTimestamp: null
-  labels:
-    controller-tools.k8s.io: "1.0"
-  name: servicepolicies.servicemesh.kubesphere.io
-spec:
-  group: servicemesh.kubesphere.io
-  names:
-    kind: ServicePolicy
-    plural: servicepolicies
-  scope: Namespaced
-  validation:
-    openAPIV3Schema:
-      properties:
-        apiVersion:
-          description: 'APIVersion defines the versioned schema of this representation
-            of an object. Servers should convert recognized schemas to the latest
-            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
-          type: string
-        kind:
-          description: 'Kind is a string value representing the REST resource this
-            object represents. Servers may infer this from the endpoint the client
-            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
-          type: string
-        metadata:
-          type: object
-        spec:
-          properties:
-            selector:
-              description: Label selector for destination rules.
-              type: object
-            template:
-              description: Template used to create a destination rule
-              properties:
-                spec:
-                  description: Spec indicates the behavior of a destination rule.
-                  properties:
-                    host:
-                      description: 'REQUIRED. The name of a service from the service
-                        registry. Service names are looked up from the platform''s
-                        service registry (e.g., Kubernetes services, Consul services,
-                        etc.) and from the hosts declared by [ServiceEntries](#ServiceEntry).
-                        Rules defined for services that do not exist in the service
-                        registry will be ignored.  *Note for Kubernetes users*: When
-                        short names are used (e.g. "reviews" instead of "reviews.default.svc.cluster.local"),
-                        Istio will interpret the short name based on the namespace
-                        of the rule, not the service. A rule in the "default" namespace
-                        containing a host "reviews will be interpreted as "reviews.default.svc.cluster.local",
-                        irrespective of the actual namespace associated with the reviews
-                        service. _To avoid potential misconfigurations, it is recommended
-                        to always use fully qualified domain names over short names._  Note
-                        that the host field applies to both HTTP and TCP services.'
-                      type: string
-                    subsets:
-                      description: One or more named sets that represent individual
-                        versions of a service. Traffic policies can be overridden
-                        at subset level.
-                      items:
-                        properties:
-                          labels:
-                            description: REQUIRED. Labels apply a filter over the
-                              endpoints of a service in the service registry. See
-                              route rules for examples of usage.
-                            type: object
-                          name:
-                            description: REQUIRED. Name of the subset. The service
-                              name and the subset name can be used for traffic splitting
-                              in a route rule.
-                            type: string
-                          trafficPolicy:
-                            description: Traffic policies that apply to this subset.
-                              Subsets inherit the traffic policies specified at the
-                              DestinationRule level. Settings specified at the subset
-                              level will override the corresponding settings specified
-                              at the DestinationRule level.
-                            properties:
-                              connectionPool:
-                                description: Settings controlling the volume of connections
-                                  to an upstream service
-                                properties:
-                                  http:
-                                    description: HTTP connection pool settings.
-                                    properties:
-                                      http1MaxPendingRequests:
-                                        description: Maximum number of pending HTTP
-                                          requests to a destination. Default 1024.
-                                        format: int32
-                                        type: integer
-                                      http2MaxRequests:
-                                        description: Maximum number of requests to
-                                          a backend. Default 1024.
-                                        format: int32
-                                        type: integer
-                                      maxRequestsPerConnection:
-                                        description: Maximum number of requests per
-                                          connection to a backend. Setting this parameter
-                                          to 1 disables keep alive.
-                                        format: int32
-                                        type: integer
-                                      maxRetries:
-                                        description: Maximum number of retries that
-                                          can be outstanding to all hosts in a cluster
-                                          at a given time. Defaults to 3.
-                                        format: int32
-                                        type: integer
-                                    type: object
-                                  tcp:
-                                    description: Settings common to both HTTP and
-                                      TCP upstream connections.
-                                    properties:
-                                      connectTimeout:
-                                        description: TCP connection timeout.
-                                        type: string
-                                      maxConnections:
-                                        description: Maximum number of HTTP1 /TCP
-                                          connections to a destination host.
-                                        format: int32
-                                        type: integer
-                                    type: object
-                                type: object
-                              loadBalancer:
-                                description: Settings controlling the load balancer
-                                  algorithms.
-                                properties:
-                                  consistentHash:
-                                    properties:
-                                      httpCookie:
-                                        description: Hash based on HTTP cookie.
-                                        properties:
-                                          name:
-                                            description: REQUIRED. Name of the cookie.
-                                            type: string
-                                          path:
-                                            description: Path to set for the cookie.
-                                            type: string
-                                          ttl:
-                                            description: REQUIRED. Lifetime of the
-                                              cookie.
-                                            type: string
-                                        required:
-                                        - name
-                                        - ttl
-                                        type: object
-                                      httpHeaderName:
-                                        description: 'It is required to specify exactly
-                                          one of the fields as hash key: HttpHeaderName,
-                                          HttpCookie, or UseSourceIP. Hash based on
-                                          a specific HTTP header.'
-                                        type: string
-                                      minimumRingSize:
-                                        description: The minimum number of virtual
-                                          nodes to use for the hash ring. Defaults
-                                          to 1024. Larger ring sizes result in more
-                                          granular load distributions. If the number
-                                          of hosts in the load balancing pool is larger
-                                          than the ring size, each host will be assigned
-                                          a single virtual node.
-                                        format: int64
-                                        type: integer
-                                      useSourceIp:
-                                        description: Hash based on the source IP address.
-                                        type: boolean
-                                    type: object
-                                  simple:
-                                    description: 'It is required to specify exactly
-                                      one of the fields: Simple or ConsistentHash'
-                                    type: string
-                                type: object
-                              outlierDetection:
-                                description: Settings controlling eviction of unhealthy
-                                  hosts from the load balancing pool
-                                properties:
-                                  baseEjectionTime:
-                                    description: 'Minimum ejection duration. A host
-                                      will remain ejected for a period equal to the
-                                      product of minimum ejection duration and the
-                                      number of times the host has been ejected. This
-                                      technique allows the system to automatically
-                                      increase the ejection period for unhealthy upstream
-                                      servers. format: 1h/1m/1s/1ms. MUST BE >=1ms.
-                                      Default is 30s.'
-                                    type: string
-                                  consecutiveErrors:
-                                    description: Number of errors before a host is
-                                      ejected from the connection pool. Defaults to
-                                      5. When the upstream host is accessed over HTTP,
-                                      a 5xx return code qualifies as an error. When
-                                      the upstream host is accessed over an opaque
-                                      TCP connection, connect timeouts and connection
-                                      error/failure events qualify as an error.
-                                    format: int32
-                                    type: integer
-                                  interval:
-                                    description: 'Time interval between ejection sweep
-                                      analysis. format: 1h/1m/1s/1ms. MUST BE >=1ms.
-                                      Default is 10s.'
-                                    type: string
-                                  maxEjectionPercent:
-                                    description: Maximum % of hosts in the load balancing
-                                      pool for the upstream service that can be ejected.
-                                      Defaults to 10%.
-                                    format: int32
-                                    type: integer
-                                type: object
-                              portLevelSettings:
-                                description: Traffic policies specific to individual
-                                  ports. Note that port level settings will override
-                                  the destination-level settings. Traffic settings
-                                  specified at the destination-level will not be inherited
-                                  when overridden by port-level settings, i.e. default
-                                  values will be applied to fields omitted in port-level
-                                  traffic policies.
-                                items:
-                                  properties:
-                                    connectionPool:
-                                      description: Settings controlling the volume
-                                        of connections to an upstream service
-                                      properties:
-                                        http:
-                                          description: HTTP connection pool settings.
-                                          properties:
-                                            http1MaxPendingRequests:
-                                              description: Maximum number of pending
-                                                HTTP requests to a destination. Default
-                                                1024.
-                                              format: int32
-                                              type: integer
-                                            http2MaxRequests:
-                                              description: Maximum number of requests
-                                                to a backend. Default 1024.
-                                              format: int32
-                                              type: integer
-                                            maxRequestsPerConnection:
-                                              description: Maximum number of requests
-                                                per connection to a backend. Setting
-                                                this parameter to 1 disables keep
-                                                alive.
-                                              format: int32
-                                              type: integer
-                                            maxRetries:
-                                              description: Maximum number of retries
-                                                that can be outstanding to all hosts
-                                                in a cluster at a given time. Defaults
-                                                to 3.
-                                              format: int32
-                                              type: integer
-                                          type: object
-                                        tcp:
-                                          description: Settings common to both HTTP
-                                            and TCP upstream connections.
-                                          properties:
-                                            connectTimeout:
-                                              description: TCP connection timeout.
-                                              type: string
-                                            maxConnections:
-                                              description: Maximum number of HTTP1
-                                                /TCP connections to a destination
-                                                host.
-                                              format: int32
-                                              type: integer
-                                          type: object
-                                      type: object
-                                    loadBalancer:
-                                      description: Settings controlling the load balancer
-                                        algorithms.
-                                      properties:
-                                        consistentHash:
-                                          properties:
-                                            httpCookie:
-                                              description: Hash based on HTTP cookie.
-                                              properties:
-                                                name:
-                                                  description: REQUIRED. Name of the
-                                                    cookie.
-                                                  type: string
-                                                path:
-                                                  description: Path to set for the
-                                                    cookie.
-                                                  type: string
-                                                ttl:
-                                                  description: REQUIRED. Lifetime
-                                                    of the cookie.
-                                                  type: string
-                                              required:
-                                              - name
-                                              - ttl
-                                              type: object
-                                            httpHeaderName:
-                                              description: 'It is required to specify
-                                                exactly one of the fields as hash
-                                                key: HttpHeaderName, HttpCookie, or
-                                                UseSourceIP. Hash based on a specific
-                                                HTTP header.'
-                                              type: string
-                                            minimumRingSize:
-                                              description: The minimum number of virtual
-                                                nodes to use for the hash ring. Defaults
-                                                to 1024. Larger ring sizes result
-                                                in more granular load distributions.
-                                                If the number of hosts in the load
-                                                balancing pool is larger than the
-                                                ring size, each host will be assigned
-                                                a single virtual node.
-                                              format: int64
-                                              type: integer
-                                            useSourceIp:
-                                              description: Hash based on the source
-                                                IP address.
-                                              type: boolean
-                                          type: object
-                                        simple:
-                                          description: 'It is required to specify
-                                            exactly one of the fields: Simple or ConsistentHash'
-                                          type: string
-                                      type: object
-                                    outlierDetection:
-                                      description: Settings controlling eviction of
-                                        unhealthy hosts from the load balancing pool
-                                      properties:
-                                        baseEjectionTime:
-                                          description: 'Minimum ejection duration.
-                                            A host will remain ejected for a period
-                                            equal to the product of minimum ejection
-                                            duration and the number of times the host
-                                            has been ejected. This technique allows
-                                            the system to automatically increase the
-                                            ejection period for unhealthy upstream
-                                            servers. format: 1h/1m/1s/1ms. MUST BE
-                                            >=1ms. Default is 30s.'
-                                          type: string
-                                        consecutiveErrors:
-                                          description: Number of errors before a host
-                                            is ejected from the connection pool. Defaults
-                                            to 5. When the upstream host is accessed
-                                            over HTTP, a 5xx return code qualifies
-                                            as an error. When the upstream host is
-                                            accessed over an opaque TCP connection,
-                                            connect timeouts and connection error/failure
-                                            events qualify as an error.
-                                          format: int32
-                                          type: integer
-                                        interval:
-                                          description: 'Time interval between ejection
-                                            sweep analysis. format: 1h/1m/1s/1ms.
-                                            MUST BE >=1ms. Default is 10s.'
-                                          type: string
-                                        maxEjectionPercent:
-                                          description: Maximum % of hosts in the load
-                                            balancing pool for the upstream service
-                                            that can be ejected. Defaults to 10%.
-                                          format: int32
-                                          type: integer
-                                      type: object
-                                    port:
-                                      description: Specifies the port name or number
-                                        of a port on the destination service on which
-                                        this policy is being applied.  Names must
-                                        comply with DNS label syntax (rfc1035) and
-                                        therefore cannot collide with numbers. If
-                                        there are multiple ports on a service with
-                                        the same protocol the names should be of the
-                                        form <protocol-name>-<DNS label>.
-                                      properties:
-                                        name:
-                                          description: Valid port name
-                                          type: string
-                                        number:
-                                          description: Valid port number
-                                          format: int32
-                                          type: integer
-                                      type: object
-                                    tls:
-                                      description: TLS related settings for connections
-                                        to the upstream service.
-                                      properties:
-                                        caCertificates:
-                                          description: 'OPTIONAL: The path to the
-                                            file containing certificate authority
-                                            certificates to use in verifying a presented
-                                            server certificate. If omitted, the proxy
-                                            will not verify the server''s certificate.
-                                            Should be empty if mode is `ISTIO_MUTUAL`.'
-                                          type: string
-                                        clientCertificate:
-                                          description: REQUIRED if mode is `MUTUAL`.
-                                            The path to the file holding the client-side
-                                            TLS certificate to use. Should be empty
-                                            if mode is `ISTIO_MUTUAL`.
-                                          type: string
-                                        mode:
-                                          description: 'REQUIRED: Indicates whether
-                                            connections to this port should be secured
-                                            using TLS. The value of this field determines
-                                            how TLS is enforced.'
-                                          type: string
-                                        privateKey:
-                                          description: REQUIRED if mode is `MUTUAL`.
-                                            The path to the file holding the client's
-                                            private key. Should be empty if mode is
-                                            `ISTIO_MUTUAL`.
-                                          type: string
-                                        sni:
-                                          description: SNI string to present to the
-                                            server during TLS handshake. Should be
-                                            empty if mode is `ISTIO_MUTUAL`.
-                                          type: string
-                                        subjectAltNames:
-                                          description: A list of alternate names to
-                                            verify the subject identity in the certificate.
-                                            If specified, the proxy will verify that
-                                            the server certificate's subject alt name
-                                            matches one of the specified values. Should
-                                            be empty if mode is `ISTIO_MUTUAL`.
-                                          items:
-                                            type: string
-                                          type: array
-                                      required:
-                                      - mode
-                                      type: object
-                                  required:
-                                  - port
-                                  type: object
-                                type: array
-                              tls:
-                                description: TLS related settings for connections
-                                  to the upstream service.
-                                properties:
-                                  caCertificates:
-                                    description: 'OPTIONAL: The path to the file containing
-                                      certificate authority certificates to use in
-                                      verifying a presented server certificate. If
-                                      omitted, the proxy will not verify the server''s
-                                      certificate. Should be empty if mode is `ISTIO_MUTUAL`.'
-                                    type: string
-                                  clientCertificate:
-                                    description: REQUIRED if mode is `MUTUAL`. The
-                                      path to the file holding the client-side TLS
-                                      certificate to use. Should be empty if mode
-                                      is `ISTIO_MUTUAL`.
-                                    type: string
-                                  mode:
-                                    description: 'REQUIRED: Indicates whether connections
-                                      to this port should be secured using TLS. The
-                                      value of this field determines how TLS is enforced.'
-                                    type: string
-                                  privateKey:
-                                    description: REQUIRED if mode is `MUTUAL`. The
-                                      path to the file holding the client's private
-                                      key. Should be empty if mode is `ISTIO_MUTUAL`.
-                                    type: string
-                                  sni:
-                                    description: SNI string to present to the server
-                                      during TLS handshake. Should be empty if mode
-                                      is `ISTIO_MUTUAL`.
-                                    type: string
-                                  subjectAltNames:
-                                    description: A list of alternate names to verify
-                                      the subject identity in the certificate. If
-                                      specified, the proxy will verify that the server
-                                      certificate's subject alt name matches one of
-                                      the specified values. Should be empty if mode
-                                      is `ISTIO_MUTUAL`.
-                                    items:
-                                      type: string
-                                    type: array
-                                required:
-                                - mode
-                                type: object
-                            type: object
-                        required:
-                        - name
-                        - labels
-                        type: object
-                      type: array
-                    trafficPolicy:
-                      description: Traffic policies to apply (load balancing policy,
-                        connection pool sizes, outlier detection).
-                      properties:
-                        connectionPool:
-                          description: Settings controlling the volume of connections
-                            to an upstream service
-                          properties:
-                            http:
-                              description: HTTP connection pool settings.
-                              properties:
-                                http1MaxPendingRequests:
-                                  description: Maximum number of pending HTTP requests
-                                    to a destination. Default 1024.
-                                  format: int32
-                                  type: integer
-                                http2MaxRequests:
-                                  description: Maximum number of requests to a backend.
-                                    Default 1024.
-                                  format: int32
-                                  type: integer
-                                maxRequestsPerConnection:
-                                  description: Maximum number of requests per connection
-                                    to a backend. Setting this parameter to 1 disables
-                                    keep alive.
-                                  format: int32
-                                  type: integer
-                                maxRetries:
-                                  description: Maximum number of retries that can
-                                    be outstanding to all hosts in a cluster at a
-                                    given time. Defaults to 3.
-                                  format: int32
-                                  type: integer
-                              type: object
-                            tcp:
-                              description: Settings common to both HTTP and TCP upstream
-                                connections.
-                              properties:
-                                connectTimeout:
-                                  description: TCP connection timeout.
-                                  type: string
-                                maxConnections:
-                                  description: Maximum number of HTTP1 /TCP connections
-                                    to a destination host.
-                                  format: int32
-                                  type: integer
-                              type: object
-                          type: object
-                        loadBalancer:
-                          description: Settings controlling the load balancer algorithms.
-                          properties:
-                            consistentHash:
-                              properties:
-                                httpCookie:
-                                  description: Hash based on HTTP cookie.
-                                  properties:
-                                    name:
-                                      description: REQUIRED. Name of the cookie.
-                                      type: string
-                                    path:
-                                      description: Path to set for the cookie.
-                                      type: string
-                                    ttl:
-                                      description: REQUIRED. Lifetime of the cookie.
-                                      type: string
-                                  required:
-                                  - name
-                                  - ttl
-                                  type: object
-                                httpHeaderName:
-                                  description: 'It is required to specify exactly
-                                    one of the fields as hash key: HttpHeaderName,
-                                    HttpCookie, or UseSourceIP. Hash based on a specific
-                                    HTTP header.'
-                                  type: string
-                                minimumRingSize:
-                                  description: The minimum number of virtual nodes
-                                    to use for the hash ring. Defaults to 1024. Larger
-                                    ring sizes result in more granular load distributions.
-                                    If the number of hosts in the load balancing pool
-                                    is larger than the ring size, each host will be
-                                    assigned a single virtual node.
-                                  format: int64
-                                  type: integer
-                                useSourceIp:
-                                  description: Hash based on the source IP address.
-                                  type: boolean
-                              type: object
-                            simple:
-                              description: 'It is required to specify exactly one
-                                of the fields: Simple or ConsistentHash'
-                              type: string
-                          type: object
-                        outlierDetection:
-                          description: Settings controlling eviction of unhealthy
-                            hosts from the load balancing pool
-                          properties:
-                            baseEjectionTime:
-                              description: 'Minimum ejection duration. A host will
-                                remain ejected for a period equal to the product of
-                                minimum ejection duration and the number of times
-                                the host has been ejected. This technique allows the
-                                system to automatically increase the ejection period
-                                for unhealthy upstream servers. format: 1h/1m/1s/1ms.
-                                MUST BE >=1ms. Default is 30s.'
-                              type: string
-                            consecutiveErrors:
-                              description: Number of errors before a host is ejected
-                                from the connection pool. Defaults to 5. When the
-                                upstream host is accessed over HTTP, a 5xx return
-                                code qualifies as an error. When the upstream host
-                                is accessed over an opaque TCP connection, connect
-                                timeouts and connection error/failure events qualify
-                                as an error.
-                              format: int32
-                              type: integer
-                            interval:
-                              description: 'Time interval between ejection sweep analysis.
-                                format: 1h/1m/1s/1ms. MUST BE >=1ms. Default is 10s.'
-                              type: string
-                            maxEjectionPercent:
-                              description: Maximum % of hosts in the load balancing
-                                pool for the upstream service that can be ejected.
-                                Defaults to 10%.
-                              format: int32
-                              type: integer
-                          type: object
-                        portLevelSettings:
-                          description: Traffic policies specific to individual ports.
-                            Note that port level settings will override the destination-level
-                            settings. Traffic settings specified at the destination-level
-                            will not be inherited when overridden by port-level settings,
-                            i.e. default values will be applied to fields omitted
-                            in port-level traffic policies.
-                          items:
-                            properties:
-                              connectionPool:
-                                description: Settings controlling the volume of connections
-                                  to an upstream service
-                                properties:
-                                  http:
-                                    description: HTTP connection pool settings.
-                                    properties:
-                                      http1MaxPendingRequests:
-                                        description: Maximum number of pending HTTP
-                                          requests to a destination. Default 1024.
-                                        format: int32
-                                        type: integer
-                                      http2MaxRequests:
-                                        description: Maximum number of requests to
-                                          a backend. Default 1024.
-                                        format: int32
-                                        type: integer
-                                      maxRequestsPerConnection:
-                                        description: Maximum number of requests per
-                                          connection to a backend. Setting this parameter
-                                          to 1 disables keep alive.
-                                        format: int32
-                                        type: integer
-                                      maxRetries:
-                                        description: Maximum number of retries that
-                                          can be outstanding to all hosts in a cluster
-                                          at a given time. Defaults to 3.
-                                        format: int32
-                                        type: integer
-                                    type: object
-                                  tcp:
-                                    description: Settings common to both HTTP and
-                                      TCP upstream connections.
-                                    properties:
-                                      connectTimeout:
-                                        description: TCP connection timeout.
-                                        type: string
-                                      maxConnections:
-                                        description: Maximum number of HTTP1 /TCP
-                                          connections to a destination host.
-                                        format: int32
-                                        type: integer
-                                    type: object
-                                type: object
-                              loadBalancer:
-                                description: Settings controlling the load balancer
-                                  algorithms.
-                                properties:
-                                  consistentHash:
-                                    properties:
-                                      httpCookie:
-                                        description: Hash based on HTTP cookie.
-                                        properties:
-                                          name:
-                                            description: REQUIRED. Name of the cookie.
-                                            type: string
-                                          path:
-                                            description: Path to set for the cookie.
-                                            type: string
-                                          ttl:
-                                            description: REQUIRED. Lifetime of the
-                                              cookie.
-                                            type: string
-                                        required:
-                                        - name
-                                        - ttl
-                                        type: object
-                                      httpHeaderName:
-                                        description: 'It is required to specify exactly
-                                          one of the fields as hash key: HttpHeaderName,
-                                          HttpCookie, or UseSourceIP. Hash based on
-                                          a specific HTTP header.'
-                                        type: string
-                                      minimumRingSize:
-                                        description: The minimum number of virtual
-                                          nodes to use for the hash ring. Defaults
-                                          to 1024. Larger ring sizes result in more
-                                          granular load distributions. If the number
-                                          of hosts in the load balancing pool is larger
-                                          than the ring size, each host will be assigned
-                                          a single virtual node.
-                                        format: int64
-                                        type: integer
-                                      useSourceIp:
-                                        description: Hash based on the source IP address.
-                                        type: boolean
-                                    type: object
-                                  simple:
-                                    description: 'It is required to specify exactly
-                                      one of the fields: Simple or ConsistentHash'
-                                    type: string
-                                type: object
-                              outlierDetection:
-                                description: Settings controlling eviction of unhealthy
-                                  hosts from the load balancing pool
-                                properties:
-                                  baseEjectionTime:
-                                    description: 'Minimum ejection duration. A host
-                                      will remain ejected for a period equal to the
-                                      product of minimum ejection duration and the
-                                      number of times the host has been ejected. This
-                                      technique allows the system to automatically
-                                      increase the ejection period for unhealthy upstream
-                                      servers. format: 1h/1m/1s/1ms. MUST BE >=1ms.
-                                      Default is 30s.'
-                                    type: string
-                                  consecutiveErrors:
-                                    description: Number of errors before a host is
-                                      ejected from the connection pool. Defaults to
-                                      5. When the upstream host is accessed over HTTP,
-                                      a 5xx return code qualifies as an error. When
-                                      the upstream host is accessed over an opaque
-                                      TCP connection, connect timeouts and connection
-                                      error/failure events qualify as an error.
-                                    format: int32
-                                    type: integer
-                                  interval:
-                                    description: 'Time interval between ejection sweep
-                                      analysis. format: 1h/1m/1s/1ms. MUST BE >=1ms.
-                                      Default is 10s.'
-                                    type: string
-                                  maxEjectionPercent:
-                                    description: Maximum % of hosts in the load balancing
-                                      pool for the upstream service that can be ejected.
-                                      Defaults to 10%.
-                                    format: int32
-                                    type: integer
-                                type: object
-                              port:
-                                description: Specifies the port name or number of
-                                  a port on the destination service on which this
-                                  policy is being applied.  Names must comply with
-                                  DNS label syntax (rfc1035) and therefore cannot
-                                  collide with numbers. If there are multiple ports
-                                  on a service with the same protocol the names should
-                                  be of the form <protocol-name>-<DNS label>.
-                                properties:
-                                  name:
-                                    description: Valid port name
-                                    type: string
-                                  number:
-                                    description: Valid port number
-                                    format: int32
-                                    type: integer
-                                type: object
-                              tls:
-                                description: TLS related settings for connections
-                                  to the upstream service.
-                                properties:
-                                  caCertificates:
-                                    description: 'OPTIONAL: The path to the file containing
-                                      certificate authority certificates to use in
-                                      verifying a presented server certificate. If
-                                      omitted, the proxy will not verify the server''s
-                                      certificate. Should be empty if mode is `ISTIO_MUTUAL`.'
-                                    type: string
-                                  clientCertificate:
-                                    description: REQUIRED if mode is `MUTUAL`. The
-                                      path to the file holding the client-side TLS
-                                      certificate to use. Should be empty if mode
-                                      is `ISTIO_MUTUAL`.
-                                    type: string
-                                  mode:
-                                    description: 'REQUIRED: Indicates whether connections
-                                      to this port should be secured using TLS. The
-                                      value of this field determines how TLS is enforced.'
-                                    type: string
-                                  privateKey:
-                                    description: REQUIRED if mode is `MUTUAL`. The
-                                      path to the file holding the client's private
-                                      key. Should be empty if mode is `ISTIO_MUTUAL`.
-                                    type: string
-                                  sni:
-                                    description: SNI string to present to the server
-                                      during TLS handshake. Should be empty if mode
-                                      is `ISTIO_MUTUAL`.
-                                    type: string
-                                  subjectAltNames:
-                                    description: A list of alternate names to verify
-                                      the subject identity in the certificate. If
-                                      specified, the proxy will verify that the server
-                                      certificate's subject alt name matches one of
-                                      the specified values. Should be empty if mode
-                                      is `ISTIO_MUTUAL`.
-                                    items:
-                                      type: string
-                                    type: array
-                                required:
-                                - mode
-                                type: object
-                            required:
-                            - port
-                            type: object
-                          type: array
-                        tls:
-                          description: TLS related settings for connections to the
-                            upstream service.
-                          properties:
-                            caCertificates:
-                              description: 'OPTIONAL: The path to the file containing
-                                certificate authority certificates to use in verifying
-                                a presented server certificate. If omitted, the proxy
-                                will not verify the server''s certificate. Should
-                                be empty if mode is `ISTIO_MUTUAL`.'
-                              type: string
-                            clientCertificate:
-                              description: REQUIRED if mode is `MUTUAL`. The path
-                                to the file holding the client-side TLS certificate
-                                to use. Should be empty if mode is `ISTIO_MUTUAL`.
-                              type: string
-                            mode:
-                              description: 'REQUIRED: Indicates whether connections
-                                to this port should be secured using TLS. The value
-                                of this field determines how TLS is enforced.'
-                              type: string
-                            privateKey:
-                              description: REQUIRED if mode is `MUTUAL`. The path
-                                to the file holding the client's private key. Should
-                                be empty if mode is `ISTIO_MUTUAL`.
-                              type: string
-                            sni:
-                              description: SNI string to present to the server during
-                                TLS handshake. Should be empty if mode is `ISTIO_MUTUAL`.
-                              type: string
-                            subjectAltNames:
-                              description: A list of alternate names to verify the
-                                subject identity in the certificate. If specified,
-                                the proxy will verify that the server certificate's
-                                subject alt name matches one of the specified values.
-                                Should be empty if mode is `ISTIO_MUTUAL`.
-                              items:
-                                type: string
-                              type: array
-                          required:
-                          - mode
-                          type: object
-                      type: object
-                  required:
-                  - host
-                  type: object
-              type: object
-          type: object
-        status:
-          type: object
-  version: v1alpha2
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crds/servicemesh_v1alpha2_strategy.yaml

@@ -1,787 +0,0 @@
-apiVersion: apiextensions.k8s.io/v1beta1
-kind: CustomResourceDefinition
-metadata:
-  creationTimestamp: null
-  labels:
-    controller-tools.k8s.io: "1.0"
-  name: strategies.servicemesh.kubesphere.io
-spec:
-  additionalPrinterColumns:
-  - JSONPath: .spec.type
-    description: type of strategy
-    name: Type
-    type: string
-  - JSONPath: .spec.template.spec.hosts
-    description: destination hosts
-    name: Hosts
-    type: string
-  - JSONPath: .metadata.creationTimestamp
-    description: 'CreationTimestamp is a timestamp representing the server time when
-      this object was created. It is not guaranteed to be set in happens-before order
-      across separate operations. Clients may not set this value. It is represented
-      in RFC3339 form and is in UTC. Populated by the system. Read-only. Null for
-      lists. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata'
-    name: Age
-    type: date
-  group: servicemesh.kubesphere.io
-  names:
-    kind: Strategy
-    plural: strategies
-  scope: Namespaced
-  validation:
-    openAPIV3Schema:
-      properties:
-        apiVersion:
-          description: 'APIVersion defines the versioned schema of this representation
-            of an object. Servers should convert recognized schemas to the latest
-            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
-          type: string
-        kind:
-          description: 'Kind is a string value representing the REST resource this
-            object represents. Servers may infer this from the endpoint the client
-            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
-          type: string
-        metadata:
-          type: object
-        spec:
-          properties:
-            governor:
-              description: Governor version, the version takes control of all incoming
-                traffic label version value
-              type: string
-            principal:
-              description: Principal version, the one as reference version label version
-                value
-              type: string
-            selector:
-              description: Label selector for virtual services.
-              type: object
-            strategyPolicy:
-              description: strategy policy, how the strategy will be applied by the
-                strategy controller
-              type: string
-            template:
-              description: Template describes the virtual service that will be created.
-              properties:
-                metadata:
-                  description: Metadata of the virtual services created from this
-                    template
-                  type: object
-                spec:
-                  description: Spec indicates the behavior of a virtual service.
-                  properties:
-                    gateways:
-                      description: The names of gateways and sidecars that should
-                        apply these routes. A single VirtualService is used for sidecars
-                        inside the mesh as well as for one or more gateways. The selection
-                        condition imposed by this field can be overridden using the
-                        source field in the match conditions of HTTP/TCP routes. The
-                        reserved word "mesh" is used to imply all the sidecars in
-                        the mesh. When this field is omitted, the default gateway
-                        ("mesh") will be used, which would apply the rule to all sidecars
-                        in the mesh. If a list of gateway names is provided, the rules
-                        will apply only to the gateways. To apply the rules to both
-                        gateways and sidecars, specify "mesh" as one of the gateway
-                        names.
-                      items:
-                        type: string
-                      type: array
-                    hosts:
-                      description: REQUIRED. The destination address for traffic captured
-                        by this virtual service. Could be a DNS name with wildcard
-                        prefix or a CIDR prefix. Depending on the platform, short-names
-                        can also be used instead of a FQDN (i.e. has no dots in the
-                        name). In such a scenario, the FQDN of the host would be derived
-                        based on the underlying platform.  For example on Kubernetes,
-                        when hosts contains a short name, Istio will interpret the
-                        short name based on the namespace of the rule. Thus, when
-                        a client namespace applies a rule in the "default" namespace
-                        containing a name "reviews, Istio will setup routes to the
-                        "reviews.default.svc.cluster.local" service. However, if a
-                        different name such as "reviews.sales.svc.cluster.local" is
-                        used, it would be treated as a FQDN during virtual host matching.
-                        In Consul, a plain service name would be resolved to the FQDN
-                        "reviews.service.consul".  Note that the hosts field applies
-                        to both HTTP and TCP services. Service inside the mesh, i.e.,
-                        those found in the service registry, must always be referred
-                        to using their alphanumeric names. IP addresses or CIDR prefixes
-                        are allowed only for services defined via the Gateway.
-                      items:
-                        type: string
-                      type: array
-                    http:
-                      description: An ordered list of route rules for HTTP traffic.
-                        The first rule matching an incoming request is used.
-                      items:
-                        properties:
-                          appendHeaders:
-                            description: Additional HTTP headers to add before forwarding
-                              a request to the destination service.
-                            type: object
-                          corsPolicy:
-                            description: Cross-Origin Resource Sharing policy
-                            properties:
-                              allowCredentials:
-                                description: Indicates whether the caller is allowed
-                                  to send the actual request (not the preflight) using
-                                  credentials. Translates to Access-Control-Allow-Credentials
-                                  header.
-                                type: boolean
-                              allowHeaders:
-                                description: List of HTTP headers that can be used
-                                  when requesting the resource. Serialized to Access-Control-Allow-Methods
-                                  header.
-                                items:
-                                  type: string
-                                type: array
-                              allowMethods:
-                                description: List of HTTP methods allowed to access
-                                  the resource. The content will be serialized into
-                                  the Access-Control-Allow-Methods header.
-                                items:
-                                  type: string
-                                type: array
-                              allowOrigin:
-                                description: The list of origins that are allowed
-                                  to perform CORS requests. The content will be serialized
-                                  into the Access-Control-Allow-Origin header. Wildcard
-                                  * will allow all origins.
-                                items:
-                                  type: string
-                                type: array
-                              exposeHeaders:
-                                description: A white list of HTTP headers that the
-                                  browsers are allowed to access. Serialized into
-                                  Access-Control-Expose-Headers header.
-                                items:
-                                  type: string
-                                type: array
-                              maxAge:
-                                description: Specifies how long the the results of
-                                  a preflight request can be cached. Translates to
-                                  the Access-Control-Max-Age header.
-                                type: string
-                            type: object
-                          fault:
-                            description: Fault injection policy to apply on HTTP traffic.
-                            properties:
-                              abort:
-                                description: Abort Http request attempts and return
-                                  error codes back to downstream service, giving the
-                                  impression that the upstream service is faulty.
-                                properties:
-                                  httpStatus:
-                                    description: REQUIRED. HTTP status code to use
-                                      to abort the Http request.
-                                    format: int64
-                                    type: integer
-                                  percent:
-                                    description: Percentage of requests to be aborted
-                                      with the error code provided (0-100).
-                                    format: int64
-                                    type: integer
-                                required:
-                                - httpStatus
-                                type: object
-                              delay:
-                                description: Delay requests before forwarding, emulating
-                                  various failures such as network issues, overloaded
-                                  upstream service, etc.
-                                properties:
-                                  exponentialDelay:
-                                    description: (-- Add a delay (based on an exponential
-                                      function) before forwarding the request. mean
-                                      delay needed to derive the exponential delay
-                                      values --)
-                                    type: string
-                                  fixedDelay:
-                                    description: 'REQUIRED. Add a fixed delay before
-                                      forwarding the request. Format: 1h/1m/1s/1ms.
-                                      MUST be >=1ms.'
-                                    type: string
-                                  percent:
-                                    description: Percentage of requests on which the
-                                      delay will be injected (0-100).
-                                    format: int64
-                                    type: integer
-                                required:
-                                - fixedDelay
-                                type: object
-                            type: object
-                          match:
-                            description: Match conditions to be satisfied for the
-                              rule to be activated. All conditions inside a single
-                              match block have AND semantics, while the list of match
-                              blocks have OR semantics. The rule is matched if any
-                              one of the match blocks succeed.
-                            items:
-                              properties:
-                                authority:
-                                  description: 'HTTP Authority values are case-sensitive
-                                    and formatted as follows:  - `exact: "value"`
-                                    for exact string match  - `prefix: "value"` for
-                                    prefix-based match  - `regex: "value"` for ECMAscript
-                                    style regex-based match'
-                                  properties:
-                                    exact:
-                                      description: exact string match
-                                      type: string
-                                    prefix:
-                                      description: prefix-based match
-                                      type: string
-                                    regex:
-                                      description: ECMAscript style regex-based match
-                                      type: string
-                                    suffix:
-                                      description: suffix-based match.
-                                      type: string
-                                  type: object
-                                gateways:
-                                  description: Names of gateways where the rule should
-                                    be applied to. Gateway names at the top of the
-                                    VirtualService (if any) are overridden. The gateway
-                                    match is independent of sourceLabels.
-                                  items:
-                                    type: string
-                                  type: array
-                                headers:
-                                  description: 'The header keys must be lowercase
-                                    and use hyphen as the separator, e.g. _x-request-id_.  Header
-                                    values are case-sensitive and formatted as follows:  -
-                                    `exact: "value"` for exact string match  - `prefix:
-                                    "value"` for prefix-based match  - `regex: "value"`
-                                    for ECMAscript style regex-based match  **Note:**
-                                    The keys `uri`, `scheme`, `method`, and `authority`
-                                    will be ignored.'
-                                  type: object
-                                method:
-                                  description: 'HTTP Method values are case-sensitive
-                                    and formatted as follows:  - `exact: "value"`
-                                    for exact string match  - `prefix: "value"` for
-                                    prefix-based match  - `regex: "value"` for ECMAscript
-                                    style regex-based match'
-                                  properties:
-                                    exact:
-                                      description: exact string match
-                                      type: string
-                                    prefix:
-                                      description: prefix-based match
-                                      type: string
-                                    regex:
-                                      description: ECMAscript style regex-based match
-                                      type: string
-                                    suffix:
-                                      description: suffix-based match.
-                                      type: string
-                                  type: object
-                                port:
-                                  description: Specifies the ports on the host that
-                                    is being addressed. Many services only expose
-                                    a single port or label ports with the protocols
-                                    they support, in these cases it is not required
-                                    to explicitly select the port.
-                                  format: int32
-                                  type: integer
-                                scheme:
-                                  description: 'URI Scheme values are case-sensitive
-                                    and formatted as follows:  - `exact: "value"`
-                                    for exact string match  - `prefix: "value"` for
-                                    prefix-based match  - `regex: "value"` for ECMAscript
-                                    style regex-based match'
-                                  properties:
-                                    exact:
-                                      description: exact string match
-                                      type: string
-                                    prefix:
-                                      description: prefix-based match
-                                      type: string
-                                    regex:
-                                      description: ECMAscript style regex-based match
-                                      type: string
-                                    suffix:
-                                      description: suffix-based match.
-                                      type: string
-                                  type: object
-                                sourceLabels:
-                                  description: One or more labels that constrain the
-                                    applicability of a rule to workloads with the
-                                    given labels. If the VirtualService has a list
-                                    of gateways specified at the top, it should include
-                                    the reserved gateway `mesh` in order for this
-                                    field to be applicable.
-                                  type: object
-                                uri:
-                                  description: 'URI to match values are case-sensitive
-                                    and formatted as follows:  - `exact: "value"`
-                                    for exact string match  - `prefix: "value"` for
-                                    prefix-based match  - `regex: "value"` for ECMAscript
-                                    style regex-based match'
-                                  properties:
-                                    exact:
-                                      description: exact string match
-                                      type: string
-                                    prefix:
-                                      description: prefix-based match
-                                      type: string
-                                    regex:
-                                      description: ECMAscript style regex-based match
-                                      type: string
-                                    suffix:
-                                      description: suffix-based match.
-                                      type: string
-                                  type: object
-                              type: object
-                            type: array
-                          mirror:
-                            description: Mirror HTTP traffic to a another destination
-                              in addition to forwarding the requests to the intended
-                              destination. Mirrored traffic is on a best effort basis
-                              where the sidecar/gateway will not wait for the mirrored
-                              cluster to respond before returning the response from
-                              the original destination.  Statistics will be generated
-                              for the mirrored destination.
-                            properties:
-                              host:
-                                description: 'REQUIRED. The name of a service from
-                                  the service registry. Service names are looked up
-                                  from the platform''s service registry (e.g., Kubernetes
-                                  services, Consul services, etc.) and from the hosts
-                                  declared by [ServiceEntry](#ServiceEntry). Traffic
-                                  forwarded to destinations that are not found in
-                                  either of the two, will be dropped.  *Note for Kubernetes
-                                  users*: When short names are used (e.g. "reviews"
-                                  instead of "reviews.default.svc.cluster.local"),
-                                  Istio will interpret the short name based on the
-                                  namespace of the rule, not the service. A rule in
-                                  the "default" namespace containing a host "reviews
-                                  will be interpreted as "reviews.default.svc.cluster.local",
-                                  irrespective of the actual namespace associated
-                                  with the reviews service. _To avoid potential misconfigurations,
-                                  it is recommended to always use fully qualified
-                                  domain names over short names._'
-                                type: string
-                              port:
-                                description: Specifies the port on the host that is
-                                  being addressed. If a service exposes only a single
-                                  port it is not required to explicitly select the
-                                  port.
-                                properties:
-                                  name:
-                                    description: Valid port name
-                                    type: string
-                                  number:
-                                    description: Valid port number
-                                    format: int32
-                                    type: integer
-                                type: object
-                              subset:
-                                description: The name of a subset within the service.
-                                  Applicable only to services within the mesh. The
-                                  subset must be defined in a corresponding DestinationRule.
-                                type: string
-                            required:
-                            - host
-                            type: object
-                          redirect:
-                            description: A http rule can either redirect or forward
-                              (default) traffic. If traffic passthrough option is
-                              specified in the rule, route/redirect will be ignored.
-                              The redirect primitive can be used to send a HTTP 302
-                              redirect to a different URI or Authority.
-                            properties:
-                              authority:
-                                description: On a redirect, overwrite the Authority/Host
-                                  portion of the URL with this value.
-                                type: string
-                              uri:
-                                description: On a redirect, overwrite the Path portion
-                                  of the URL with this value. Note that the entire
-                                  path will be replaced, irrespective of the request
-                                  URI being matched as an exact path or prefix.
-                                type: string
-                            type: object
-                          removeResponseHeaders:
-                            description: Http headers to remove before returning the
-                              response to the caller
-                            type: object
-                          retries:
-                            description: Retry policy for HTTP requests.
-                            properties:
-                              attempts:
-                                description: REQUIRED. Number of retries for a given
-                                  request. The interval between retries will be determined
-                                  automatically (25ms+). Actual number of retries
-                                  attempted depends on the httpReqTimeout.
-                                format: int64
-                                type: integer
-                              perTryTimeout:
-                                description: 'Timeout per retry attempt for a given
-                                  request. format: 1h/1m/1s/1ms. MUST BE >=1ms.'
-                                type: string
-                            required:
-                            - attempts
-                            - perTryTimeout
-                            type: object
-                          rewrite:
-                            description: Rewrite HTTP URIs and Authority headers.
-                              Rewrite cannot be used with Redirect primitive. Rewrite
-                              will be performed before forwarding.
-                            properties:
-                              authority:
-                                description: rewrite the Authority/Host header with
-                                  this value.
-                                type: string
-                              uri:
-                                description: rewrite the path (or the prefix) portion
-                                  of the URI with this value. If the original URI
-                                  was matched based on prefix, the value provided
-                                  in this field will replace the corresponding matched
-                                  prefix.
-                                type: string
-                            type: object
-                          route:
-                            description: A http rule can either redirect or forward
-                              (default) traffic. The forwarding target can be one
-                              of several versions of a service (see glossary in beginning
-                              of document). Weights associated with the service version
-                              determine the proportion of traffic it receives.
-                            items:
-                              properties:
-                                destination:
-                                  description: REQUIRED. Destination uniquely identifies
-                                    the instances of a service to which the request/connection
-                                    should be forwarded to.
-                                  properties:
-                                    host:
-                                      description: 'REQUIRED. The name of a service
-                                        from the service registry. Service names are
-                                        looked up from the platform''s service registry
-                                        (e.g., Kubernetes services, Consul services,
-                                        etc.) and from the hosts declared by [ServiceEntry](#ServiceEntry).
-                                        Traffic forwarded to destinations that are
-                                        not found in either of the two, will be dropped.  *Note
-                                        for Kubernetes users*: When short names are
-                                        used (e.g. "reviews" instead of "reviews.default.svc.cluster.local"),
-                                        Istio will interpret the short name based
-                                        on the namespace of the rule, not the service.
-                                        A rule in the "default" namespace containing
-                                        a host "reviews will be interpreted as "reviews.default.svc.cluster.local",
-                                        irrespective of the actual namespace associated
-                                        with the reviews service. _To avoid potential
-                                        misconfigurations, it is recommended to always
-                                        use fully qualified domain names over short
-                                        names._'
-                                      type: string
-                                    port:
-                                      description: Specifies the port on the host
-                                        that is being addressed. If a service exposes
-                                        only a single port it is not required to explicitly
-                                        select the port.
-                                      properties:
-                                        name:
-                                          description: Valid port name
-                                          type: string
-                                        number:
-                                          description: Valid port number
-                                          format: int32
-                                          type: integer
-                                      type: object
-                                    subset:
-                                      description: The name of a subset within the
-                                        service. Applicable only to services within
-                                        the mesh. The subset must be defined in a
-                                        corresponding DestinationRule.
-                                      type: string
-                                  required:
-                                  - host
-                                  type: object
-                                weight:
-                                  description: REQUIRED. The proportion of traffic
-                                    to be forwarded to the service version. (0-100).
-                                    Sum of weights across destinations SHOULD BE ==
-                                    100. If there is only destination in a rule, the
-                                    weight value is assumed to be 100.
-                                  format: int64
-                                  type: integer
-                              required:
-                              - destination
-                              - weight
-                              type: object
-                            type: array
-                          timeout:
-                            description: Timeout for HTTP requests.
-                            type: string
-                          websocketUpgrade:
-                            description: Indicates that a HTTP/1.1 client connection
-                              to this particular route should be allowed (and expected)
-                              to upgrade to a WebSocket connection. The default is
-                              false. Istio's reference sidecar implementation (Envoy)
-                              expects the first request to this route to contain the
-                              WebSocket upgrade headers. Otherwise, the request will
-                              be rejected. Note that Websocket allows secondary protocol
-                              negotiation which may then be subject to further routing
-                              rules based on the protocol selected.
-                            type: boolean
-                        type: object
-                      type: array
-                    tcp:
-                      description: An ordered list of route rules for TCP traffic.
-                        The first rule matching an incoming request is used.
-                      items:
-                        properties:
-                          match:
-                            description: Match conditions to be satisfied for the
-                              rule to be activated. All conditions inside a single
-                              match block have AND semantics, while the list of match
-                              blocks have OR semantics. The rule is matched if any
-                              one of the match blocks succeed.
-                            items:
-                              properties:
-                                destinationSubnets:
-                                  description: IPv4 or IPv6 ip address of destination
-                                    with optional subnet.  E.g., a.b.c.d/xx form or
-                                    just a.b.c.d.
-                                  items:
-                                    type: string
-                                  type: array
-                                gateways:
-                                  description: Names of gateways where the rule should
-                                    be applied to. Gateway names at the top of the
-                                    VirtualService (if any) are overridden. The gateway
-                                    match is independent of sourceLabels.
-                                  items:
-                                    type: string
-                                  type: array
-                                port:
-                                  description: Specifies the port on the host that
-                                    is being addressed. Many services only expose
-                                    a single port or label ports with the protocols
-                                    they support, in these cases it is not required
-                                    to explicitly select the port.
-                                  format: int64
-                                  type: integer
-                                sourceLabels:
-                                  description: One or more labels that constrain the
-                                    applicability of a rule to workloads with the
-                                    given labels. If the VirtualService has a list
-                                    of gateways specified at the top, it should include
-                                    the reserved gateway `mesh` in order for this
-                                    field to be applicable.
-                                  type: object
-                              type: object
-                            type: array
-                          route:
-                            description: The destinations to which the connection
-                              should be forwarded to. Weights must add to 100%.
-                            items:
-                              properties:
-                                destination:
-                                  description: REQUIRED. Destination uniquely identifies
-                                    the instances of a service to which the request/connection
-                                    should be forwarded to.
-                                  properties:
-                                    host:
-                                      description: 'REQUIRED. The name of a service
-                                        from the service registry. Service names are
-                                        looked up from the platform''s service registry
-                                        (e.g., Kubernetes services, Consul services,
-                                        etc.) and from the hosts declared by [ServiceEntry](#ServiceEntry).
-                                        Traffic forwarded to destinations that are
-                                        not found in either of the two, will be dropped.  *Note
-                                        for Kubernetes users*: When short names are
-                                        used (e.g. "reviews" instead of "reviews.default.svc.cluster.local"),
-                                        Istio will interpret the short name based
-                                        on the namespace of the rule, not the service.
-                                        A rule in the "default" namespace containing
-                                        a host "reviews will be interpreted as "reviews.default.svc.cluster.local",
-                                        irrespective of the actual namespace associated
-                                        with the reviews service. _To avoid potential
-                                        misconfigurations, it is recommended to always
-                                        use fully qualified domain names over short
-                                        names._'
-                                      type: string
-                                    port:
-                                      description: Specifies the port on the host
-                                        that is being addressed. If a service exposes
-                                        only a single port it is not required to explicitly
-                                        select the port.
-                                      properties:
-                                        name:
-                                          description: Valid port name
-                                          type: string
-                                        number:
-                                          description: Valid port number
-                                          format: int32
-                                          type: integer
-                                      type: object
-                                    subset:
-                                      description: The name of a subset within the
-                                        service. Applicable only to services within
-                                        the mesh. The subset must be defined in a
-                                        corresponding DestinationRule.
-                                      type: string
-                                  required:
-                                  - host
-                                  type: object
-                                weight:
-                                  description: REQUIRED. The proportion of traffic
-                                    to be forwarded to the service version. (0-100).
-                                    Sum of weights across destinations SHOULD BE ==
-                                    100. If there is only destination in a rule, the
-                                    weight value is assumed to be 100.
-                                  format: int64
-                                  type: integer
-                              required:
-                              - destination
-                              - weight
-                              type: object
-                            type: array
-                        required:
-                        - match
-                        - route
-                        type: object
-                      type: array
-                    tls:
-                      items:
-                        properties:
-                          match:
-                            description: REQUIRED. Match conditions to be satisfied
-                              for the rule to be activated. All conditions inside
-                              a single match block have AND semantics, while the list
-                              of match blocks have OR semantics. The rule is matched
-                              if any one of the match blocks succeed.
-                            items:
-                              properties:
-                                destinationSubnets:
-                                  description: IPv4 or IPv6 ip addresses of destination
-                                    with optional subnet.  E.g., a.b.c.d/xx form or
-                                    just a.b.c.d.
-                                  items:
-                                    type: string
-                                  type: array
-                                gateways:
-                                  description: Names of gateways where the rule should
-                                    be applied to. Gateway names at the top of the
-                                    VirtualService (if any) are overridden. The gateway
-                                    match is independent of sourceLabels.
-                                  items:
-                                    type: string
-                                  type: array
-                                port:
-                                  description: Specifies the port on the host that
-                                    is being addressed. Many services only expose
-                                    a single port or label ports with the protocols
-                                    they support, in these cases it is not required
-                                    to explicitly select the port.
-                                  format: int64
-                                  type: integer
-                                sniHosts:
-                                  description: REQUIRED. SNI (server name indicator)
-                                    to match on. Wildcard prefixes can be used in
-                                    the SNI value, e.g., *.com will match foo.example.com
-                                    as well as example.com. An SNI value must be a
-                                    subset (i.e., fall within the domain) of the corresponding
-                                    virtual service's hosts
-                                  items:
-                                    type: string
-                                  type: array
-                                sourceLabels:
-                                  description: One or more labels that constrain the
-                                    applicability of a rule to workloads with the
-                                    given labels. If the VirtualService has a list
-                                    of gateways specified at the top, it should include
-                                    the reserved gateway `mesh` in order for this
-                                    field to be applicable.
-                                  type: object
-                              required:
-                              - sniHosts
-                              type: object
-                            type: array
-                          route:
-                            description: The destination to which the connection should
-                              be forwarded to.
-                            items:
-                              properties:
-                                destination:
-                                  description: REQUIRED. Destination uniquely identifies
-                                    the instances of a service to which the request/connection
-                                    should be forwarded to.
-                                  properties:
-                                    host:
-                                      description: 'REQUIRED. The name of a service
-                                        from the service registry. Service names are
-                                        looked up from the platform''s service registry
-                                        (e.g., Kubernetes services, Consul services,
-                                        etc.) and from the hosts declared by [ServiceEntry](#ServiceEntry).
-                                        Traffic forwarded to destinations that are
-                                        not found in either of the two, will be dropped.  *Note
-                                        for Kubernetes users*: When short names are
-                                        used (e.g. "reviews" instead of "reviews.default.svc.cluster.local"),
-                                        Istio will interpret the short name based
-                                        on the namespace of the rule, not the service.
-                                        A rule in the "default" namespace containing
-                                        a host "reviews will be interpreted as "reviews.default.svc.cluster.local",
-                                        irrespective of the actual namespace associated
-                                        with the reviews service. _To avoid potential
-                                        misconfigurations, it is recommended to always
-                                        use fully qualified domain names over short
-                                        names._'
-                                      type: string
-                                    port:
-                                      description: Specifies the port on the host
-                                        that is being addressed. If a service exposes
-                                        only a single port it is not required to explicitly
-                                        select the port.
-                                      properties:
-                                        name:
-                                          description: Valid port name
-                                          type: string
-                                        number:
-                                          description: Valid port number
-                                          format: int32
-                                          type: integer
-                                      type: object
-                                    subset:
-                                      description: The name of a subset within the
-                                        service. Applicable only to services within
-                                        the mesh. The subset must be defined in a
-                                        corresponding DestinationRule.
-                                      type: string
-                                  required:
-                                  - host
-                                  type: object
-                                weight:
-                                  description: REQUIRED. The proportion of traffic
-                                    to be forwarded to the service version. (0-100).
-                                    Sum of weights across destinations SHOULD BE ==
-                                    100. If there is only destination in a rule, the
-                                    weight value is assumed to be 100.
-                                  format: int64
-                                  type: integer
-                              required:
-                              - destination
-                              - weight
-                              type: object
-                            type: array
-                        required:
-                        - match
-                        - route
-                        type: object
-                      type: array
-                  required:
-                  - hosts
-                  type: object
-              type: object
-            type:
-              description: Strategy type
-              type: string
-          type: object
-        status:
-          type: object
-  version: v1alpha2
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crds/snapshot.storage.k8s.io_volumesnapshot.yaml

@@ -0,0 +1,503 @@
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.2.5
+    api-approved.kubernetes.io: "https://github.com/kubernetes-csi/external-snapshotter/pull/260"
+  creationTimestamp: null
+  name: volumesnapshotclasses.snapshot.storage.k8s.io
+spec:
+  additionalPrinterColumns:
+    - JSONPath: .driver
+      name: Driver
+      type: string
+    - JSONPath: .deletionPolicy
+      description: Determines whether a VolumeSnapshotContent created through the VolumeSnapshotClass
+        should be deleted when its bound VolumeSnapshot is deleted.
+      name: DeletionPolicy
+      type: string
+    - JSONPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+  group: snapshot.storage.k8s.io
+  names:
+    kind: VolumeSnapshotClass
+    listKind: VolumeSnapshotClassList
+    plural: volumesnapshotclasses
+    singular: volumesnapshotclass
+  preserveUnknownFields: false
+  scope: Cluster
+  subresources: {}
+  validation:
+    openAPIV3Schema:
+      description: VolumeSnapshotClass specifies parameters that a underlying storage
+        system uses when creating a volume snapshot. A specific VolumeSnapshotClass
+        is used by specifying its name in a VolumeSnapshot object. VolumeSnapshotClasses
+        are non-namespaced
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        deletionPolicy:
+          description: deletionPolicy determines whether a VolumeSnapshotContent created
+            through the VolumeSnapshotClass should be deleted when its bound VolumeSnapshot
+            is deleted. Supported values are "Retain" and "Delete". "Retain" means
+            that the VolumeSnapshotContent and its physical snapshot on underlying
+            storage system are kept. "Delete" means that the VolumeSnapshotContent
+            and its physical snapshot on underlying storage system are deleted. Required.
+          enum:
+            - Delete
+            - Retain
+          type: string
+        driver:
+          description: driver is the name of the storage driver that handles this
+            VolumeSnapshotClass. Required.
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        parameters:
+          additionalProperties:
+            type: string
+          description: parameters is a key-value map with storage driver specific
+            parameters for creating snapshots. These values are opaque to Kubernetes.
+          type: object
+      required:
+        - deletionPolicy
+        - driver
+      type: object
+  version: v1beta1
+  versions:
+    - name: v1beta1
+      served: true
+      storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.2.5
+    api-approved.kubernetes.io: "https://github.com/kubernetes-csi/external-snapshotter/pull/260"
+  creationTimestamp: null
+  name: volumesnapshotcontents.snapshot.storage.k8s.io
+spec:
+  additionalPrinterColumns:
+    - JSONPath: .status.readyToUse
+      description: Indicates if a snapshot is ready to be used to restore a volume.
+      name: ReadyToUse
+      type: boolean
+    - JSONPath: .status.restoreSize
+      description: Represents the complete size of the snapshot in bytes
+      name: RestoreSize
+      type: integer
+    - JSONPath: .spec.deletionPolicy
+      description: Determines whether this VolumeSnapshotContent and its physical snapshot
+        on the underlying storage system should be deleted when its bound VolumeSnapshot
+        is deleted.
+      name: DeletionPolicy
+      type: string
+    - JSONPath: .spec.driver
+      description: Name of the CSI driver used to create the physical snapshot on the
+        underlying storage system.
+      name: Driver
+      type: string
+    - JSONPath: .spec.volumeSnapshotClassName
+      description: Name of the VolumeSnapshotClass to which this snapshot belongs.
+      name: VolumeSnapshotClass
+      type: string
+    - JSONPath: .spec.volumeSnapshotRef.name
+      description: Name of the VolumeSnapshot object to which this VolumeSnapshotContent
+        object is bound.
+      name: VolumeSnapshot
+      type: string
+    - JSONPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+  group: snapshot.storage.k8s.io
+  names:
+    kind: VolumeSnapshotContent
+    listKind: VolumeSnapshotContentList
+    plural: volumesnapshotcontents
+    singular: volumesnapshotcontent
+  preserveUnknownFields: false
+  scope: Cluster
+  subresources:
+    status: {}
+  validation:
+    openAPIV3Schema:
+      description: VolumeSnapshotContent represents the actual "on-disk" snapshot
+        object in the underlying storage system
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        spec:
+          description: spec defines properties of a VolumeSnapshotContent created
+            by the underlying storage system. Required.
+          properties:
+            deletionPolicy:
+              description: deletionPolicy determines whether this VolumeSnapshotContent
+                and its physical snapshot on the underlying storage system should
+                be deleted when its bound VolumeSnapshot is deleted. Supported values
+                are "Retain" and "Delete". "Retain" means that the VolumeSnapshotContent
+                and its physical snapshot on underlying storage system are kept. "Delete"
+                means that the VolumeSnapshotContent and its physical snapshot on
+                underlying storage system are deleted. In dynamic snapshot creation
+                case, this field will be filled in with the "DeletionPolicy" field
+                defined in the VolumeSnapshotClass the VolumeSnapshot refers to. For
+                pre-existing snapshots, users MUST specify this field when creating
+                the VolumeSnapshotContent object. Required.
+              enum:
+                - Delete
+                - Retain
+              type: string
+            driver:
+              description: driver is the name of the CSI driver used to create the
+                physical snapshot on the underlying storage system. This MUST be the
+                same as the name returned by the CSI GetPluginName() call for that
+                driver. Required.
+              type: string
+            source:
+              description: source specifies from where a snapshot will be created.
+                This field is immutable after creation. Required.
+              properties:
+                snapshotHandle:
+                  description: snapshotHandle specifies the CSI "snapshot_id" of a
+                    pre-existing snapshot on the underlying storage system. This field
+                    is immutable.
+                  type: string
+                volumeHandle:
+                  description: volumeHandle specifies the CSI "volume_id" of the volume
+                    from which a snapshot should be dynamically taken from. This field
+                    is immutable.
+                  type: string
+              type: object
+            volumeSnapshotClassName:
+              description: name of the VolumeSnapshotClass to which this snapshot
+                belongs.
+              type: string
+            volumeSnapshotRef:
+              description: volumeSnapshotRef specifies the VolumeSnapshot object to
+                which this VolumeSnapshotContent object is bound. VolumeSnapshot.Spec.VolumeSnapshotContentName
+                field must reference to this VolumeSnapshotContent's name for the
+                bidirectional binding to be valid. For a pre-existing VolumeSnapshotContent
+                object, name and namespace of the VolumeSnapshot object MUST be provided
+                for binding to happen. This field is immutable after creation. Required.
+              properties:
+                apiVersion:
+                  description: API version of the referent.
+                  type: string
+                fieldPath:
+                  description: 'If referring to a piece of an object instead of an
+                    entire object, this string should contain a valid JSON/Go field
+                    access statement, such as desiredState.manifest.containers[2].
+                    For example, if the object reference is to a container within
+                    a pod, this would take on a value like: "spec.containers{name}"
+                    (where "name" refers to the name of the container that triggered
+                    the event) or if no container name is specified "spec.containers[2]"
+                    (container with index 2 in this pod). This syntax is chosen only
+                    to have some well-defined way of referencing a part of an object.
+                    TODO: this design is not final and this field is subject to change
+                    in the future.'
+                  type: string
+                kind:
+                  description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                  type: string
+                name:
+                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                  type: string
+                namespace:
+                  description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                  type: string
+                resourceVersion:
+                  description: 'Specific resourceVersion to which this reference is
+                    made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                  type: string
+                uid:
+                  description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                  type: string
+              type: object
+          required:
+            - deletionPolicy
+            - driver
+            - source
+            - volumeSnapshotRef
+          type: object
+        status:
+          description: status represents the current information of a snapshot.
+          properties:
+            creationTime:
+              description: creationTime is the timestamp when the point-in-time snapshot
+                is taken by the underlying storage system. In dynamic snapshot creation
+                case, this field will be filled in with the "creation_time" value
+                returned from CSI "CreateSnapshotRequest" gRPC call. For a pre-existing
+                snapshot, this field will be filled with the "creation_time" value
+                returned from the CSI "ListSnapshots" gRPC call if the driver supports
+                it. If not specified, it indicates the creation time is unknown. The
+                format of this field is a Unix nanoseconds time encoded as an int64.
+                On Unix, the command `date +%s%N` returns the current time in nanoseconds
+                since 1970-01-01 00:00:00 UTC.
+              format: int64
+              type: integer
+            error:
+              description: error is the latest observed error during snapshot creation,
+                if any.
+              properties:
+                message:
+                  description: 'message is a string detailing the encountered error
+                    during snapshot creation if specified. NOTE: message may be logged,
+                    and it should not contain sensitive information.'
+                  type: string
+                time:
+                  description: time is the timestamp when the error was encountered.
+                  format: date-time
+                  type: string
+              type: object
+            readyToUse:
+              description: readyToUse indicates if a snapshot is ready to be used
+                to restore a volume. In dynamic snapshot creation case, this field
+                will be filled in with the "ready_to_use" value returned from CSI
+                "CreateSnapshotRequest" gRPC call. For a pre-existing snapshot, this
+                field will be filled with the "ready_to_use" value returned from the
+                CSI "ListSnapshots" gRPC call if the driver supports it, otherwise,
+                this field will be set to "True". If not specified, it means the readiness
+                of a snapshot is unknown.
+              type: boolean
+            restoreSize:
+              description: restoreSize represents the complete size of the snapshot
+                in bytes. In dynamic snapshot creation case, this field will be filled
+                in with the "size_bytes" value returned from CSI "CreateSnapshotRequest"
+                gRPC call. For a pre-existing snapshot, this field will be filled
+                with the "size_bytes" value returned from the CSI "ListSnapshots"
+                gRPC call if the driver supports it. When restoring a volume from
+                this snapshot, the size of the volume MUST NOT be smaller than the
+                restoreSize if it is specified, otherwise the restoration will fail.
+                If not specified, it indicates that the size is unknown.
+              format: int64
+              minimum: 0
+              type: integer
+            snapshotHandle:
+              description: snapshotHandle is the CSI "snapshot_id" of a snapshot on
+                the underlying storage system. If not specified, it indicates that
+                dynamic snapshot creation has either failed or it is still in progress.
+              type: string
+          type: object
+      required:
+        - spec
+      type: object
+  version: v1beta1
+  versions:
+    - name: v1beta1
+      served: true
+      storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.2.5
+    api-approved.kubernetes.io: "https://github.com/kubernetes-csi/external-snapshotter/pull/260"
+  creationTimestamp: null
+  name: volumesnapshots.snapshot.storage.k8s.io
+spec:
+  additionalPrinterColumns:
+    - JSONPath: .status.readyToUse
+      description: Indicates if a snapshot is ready to be used to restore a volume.
+      name: ReadyToUse
+      type: boolean
+    - JSONPath: .spec.source.persistentVolumeClaimName
+      description: Name of the source PVC from where a dynamically taken snapshot will
+        be created.
+      name: SourcePVC
+      type: string
+    - JSONPath: .spec.source.volumeSnapshotContentName
+      description: Name of the VolumeSnapshotContent which represents a pre-provisioned
+        snapshot.
+      name: SourceSnapshotContent
+      type: string
+    - JSONPath: .status.restoreSize
+      description: Represents the complete size of the snapshot.
+      name: RestoreSize
+      type: string
+    - JSONPath: .spec.volumeSnapshotClassName
+      description: The name of the VolumeSnapshotClass requested by the VolumeSnapshot.
+      name: SnapshotClass
+      type: string
+    - JSONPath: .status.boundVolumeSnapshotContentName
+      description: The name of the VolumeSnapshotContent to which this VolumeSnapshot
+        is bound.
+      name: SnapshotContent
+      type: string
+    - JSONPath: .status.creationTime
+      description: Timestamp when the point-in-time snapshot is taken by the underlying
+        storage system.
+      name: CreationTime
+      type: date
+    - JSONPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+  group: snapshot.storage.k8s.io
+  names:
+    kind: VolumeSnapshot
+    listKind: VolumeSnapshotList
+    plural: volumesnapshots
+    singular: volumesnapshot
+  preserveUnknownFields: false
+  scope: Namespaced
+  subresources:
+    status: {}
+  validation:
+    openAPIV3Schema:
+      description: VolumeSnapshot is a user's request for either creating a point-in-time
+        snapshot of a persistent volume, or binding to a pre-existing snapshot.
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+          type: string
+        spec:
+          description: 'spec defines the desired characteristics of a snapshot requested
+            by a user. More info: https://kubernetes.io/docs/concepts/storage/volume-snapshots#volumesnapshots
+            Required.'
+          properties:
+            source:
+              description: source specifies where a snapshot will be created from.
+                This field is immutable after creation. Required.
+              properties:
+                persistentVolumeClaimName:
+                  description: persistentVolumeClaimName specifies the name of the
+                    PersistentVolumeClaim object in the same namespace as the VolumeSnapshot
+                    object where the snapshot should be dynamically taken from. This
+                    field is immutable.
+                  type: string
+                volumeSnapshotContentName:
+                  description: volumeSnapshotContentName specifies the name of a pre-existing
+                    VolumeSnapshotContent object. This field is immutable.
+                  type: string
+              type: object
+            volumeSnapshotClassName:
+              description: 'volumeSnapshotClassName is the name of the VolumeSnapshotClass
+                requested by the VolumeSnapshot. If not specified, the default snapshot
+                class will be used if one exists. If not specified, and there is no
+                default snapshot class, dynamic snapshot creation will fail. Empty
+                string is not allowed for this field. TODO(xiangqian): a webhook validation
+                on empty string. More info: https://kubernetes.io/docs/concepts/storage/volume-snapshot-classes'
+              type: string
+          required:
+            - source
+          type: object
+        status:
+          description: 'status represents the current information of a snapshot. NOTE:
+            status can be modified by sources other than system controllers, and must
+            not be depended upon for accuracy. Controllers should only use information
+            from the VolumeSnapshotContent object after verifying that the binding
+            is accurate and complete.'
+          properties:
+            boundVolumeSnapshotContentName:
+              description: 'boundVolumeSnapshotContentName represents the name of
+                the VolumeSnapshotContent object to which the VolumeSnapshot object
+                is bound. If not specified, it indicates that the VolumeSnapshot object
+                has not been successfully bound to a VolumeSnapshotContent object
+                yet. NOTE: Specified boundVolumeSnapshotContentName alone does not
+                mean binding       is valid. Controllers MUST always verify bidirectional
+                binding between       VolumeSnapshot and VolumeSnapshotContent to
+                avoid possible security issues.'
+              type: string
+            creationTime:
+              description: creationTime is the timestamp when the point-in-time snapshot
+                is taken by the underlying storage system. In dynamic snapshot creation
+                case, this field will be filled in with the "creation_time" value
+                returned from CSI "CreateSnapshotRequest" gRPC call. For a pre-existing
+                snapshot, this field will be filled with the "creation_time" value
+                returned from the CSI "ListSnapshots" gRPC call if the driver supports
+                it. If not specified, it indicates that the creation time of the snapshot
+                is unknown.
+              format: date-time
+              type: string
+            error:
+              description: error is the last observed error during snapshot creation,
+                if any. This field could be helpful to upper level controllers(i.e.,
+                application controller) to decide whether they should continue on
+                waiting for the snapshot to be created based on the type of error
+                reported.
+              properties:
+                message:
+                  description: 'message is a string detailing the encountered error
+                    during snapshot creation if specified. NOTE: message may be logged,
+                    and it should not contain sensitive information.'
+                  type: string
+                time:
+                  description: time is the timestamp when the error was encountered.
+                  format: date-time
+                  type: string
+              type: object
+            readyToUse:
+              description: readyToUse indicates if a snapshot is ready to be used
+                to restore a volume. In dynamic snapshot creation case, this field
+                will be filled in with the "ready_to_use" value returned from CSI
+                "CreateSnapshotRequest" gRPC call. For a pre-existing snapshot, this
+                field will be filled with the "ready_to_use" value returned from the
+                CSI "ListSnapshots" gRPC call if the driver supports it, otherwise,
+                this field will be set to "True". If not specified, it means the readiness
+                of a snapshot is unknown.
+              type: boolean
+            restoreSize:
+              anyOf:
+                - type: integer
+                - type: string
+              description: restoreSize represents the complete size of the snapshot
+                in bytes. In dynamic snapshot creation case, this field will be filled
+                in with the "size_bytes" value returned from CSI "CreateSnapshotRequest"
+                gRPC call. For a pre-existing snapshot, this field will be filled
+                with the "size_bytes" value returned from the CSI "ListSnapshots"
+                gRPC call if the driver supports it. When restoring a volume from
+                this snapshot, the size of the volume MUST NOT be smaller than the
+                restoreSize if it is specified, otherwise the restoration will fail.
+                If not specified, it indicates that the size is unknown.
+              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+              x-kubernetes-int-or-string: true
+          type: object
+      required:
+        - spec
+      type: object
+  version: v1beta1
+  versions:
+    - name: v1beta1
+      served: true
+      storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/storage.kubesphere.io_provisionercapabilities.yaml

@@ -0,0 +1,120 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: provisionercapabilities.storage.kubesphere.io
+spec:
+  group: storage.kubesphere.io
+  names:
+    kind: ProvisionerCapability
+    listKind: ProvisionerCapabilityList
+    plural: provisionercapabilities
+    singular: provisionercapability
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.pluginInfo.name
+      name: Provisioner
+      type: string
+    - jsonPath: .spec.features.volume.expandMode
+      name: Expand
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: ProvisionerCapability is the schema for the provisionercapability
+          API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: ProvisionerCapabilitySpec defines the desired state of ProvisionerCapability
+            properties:
+              features:
+                description: CapabilityFeatures describe storage features
+                properties:
+                  snapshot:
+                    description: SnapshotFeature describe snapshot features
+                    properties:
+                      create:
+                        type: boolean
+                      list:
+                        type: boolean
+                    required:
+                    - create
+                    - list
+                    type: object
+                  topology:
+                    type: boolean
+                  volume:
+                    description: VolumeFeature describe volume features
+                    properties:
+                      attach:
+                        type: boolean
+                      clone:
+                        type: boolean
+                      create:
+                        type: boolean
+                      expandMode:
+                        type: string
+                      list:
+                        type: boolean
+                      stats:
+                        type: boolean
+                    required:
+                    - attach
+                    - clone
+                    - create
+                    - expandMode
+                    - list
+                    - stats
+                    type: object
+                required:
+                - snapshot
+                - topology
+                - volume
+                type: object
+              pluginInfo:
+                description: PluginInfo describes plugin info
+                properties:
+                  name:
+                    type: string
+                  version:
+                    type: string
+                required:
+                - name
+                - version
+                type: object
+            required:
+            - features
+            - pluginInfo
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/storage.kubesphere.io_storageclasscapabilities.yaml

@@ -0,0 +1,120 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: storageclasscapabilities.storage.kubesphere.io
+spec:
+  group: storage.kubesphere.io
+  names:
+    kind: StorageClassCapability
+    listKind: StorageClassCapabilityList
+    plural: storageclasscapabilities
+    singular: storageclasscapability
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.provisioner
+      name: Provisioner
+      type: string
+    - jsonPath: .spec.features.volume.create
+      name: Volume
+      type: boolean
+    - jsonPath: .spec.features.volume.expandMode
+      name: Expand
+      type: string
+    - jsonPath: .spec.features.volume.clone
+      name: Clone
+      type: boolean
+    - jsonPath: .spec.features.snapshot.create
+      name: Snapshot
+      type: boolean
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: StorageClassCapability is the Schema for the storage class capability
+          API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: StorageClassCapabilitySpec defines the desired state of StorageClassCapability
+            properties:
+              features:
+                description: CapabilityFeatures describe storage features
+                properties:
+                  snapshot:
+                    description: SnapshotFeature describe snapshot features
+                    properties:
+                      create:
+                        type: boolean
+                      list:
+                        type: boolean
+                    required:
+                    - create
+                    - list
+                    type: object
+                  topology:
+                    type: boolean
+                  volume:
+                    description: VolumeFeature describe volume features
+                    properties:
+                      attach:
+                        type: boolean
+                      clone:
+                        type: boolean
+                      create:
+                        type: boolean
+                      expandMode:
+                        type: string
+                      list:
+                        type: boolean
+                      stats:
+                        type: boolean
+                    required:
+                    - attach
+                    - clone
+                    - create
+                    - expandMode
+                    - list
+                    - stats
+                    type: object
+                required:
+                - snapshot
+                - topology
+                - volume
+                type: object
+              provisioner:
+                type: string
+            required:
+            - features
+            - provisioner
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/tenant.kubesphere.io_workspaces.yaml

@@ -0,0 +1,53 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: workspaces.tenant.kubesphere.io
+spec:
+  group: tenant.kubesphere.io
+  names:
+    categories:
+    - tenant
+    kind: Workspace
+    listKind: WorkspaceList
+    plural: workspaces
+    singular: workspace
+  scope: Cluster
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: Workspace is the Schema for the workspaces API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: WorkspaceSpec defines the desired state of Workspace
+            properties:
+              manager:
+                type: string
+              networkIsolation:
+                type: boolean
+            type: object
+          status:
+            description: WorkspaceStatus defines the observed state of Workspace
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/tenant.kubesphere.io_workspacetemplates.yaml

@@ -0,0 +1,125 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: workspacetemplates.tenant.kubesphere.io
+spec:
+  group: tenant.kubesphere.io
+  names:
+    categories:
+    - tenant
+    kind: WorkspaceTemplate
+    listKind: WorkspaceTemplateList
+    plural: workspacetemplates
+    singular: workspacetemplate
+  scope: Cluster
+  versions:
+  - name: v1alpha2
+    schema:
+      openAPIV3Schema:
+        description: WorkspaceTemplate is the Schema for the workspacetemplates API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              overrides:
+                items:
+                  properties:
+                    clusterName:
+                      type: string
+                    clusterOverrides:
+                      items:
+                        properties:
+                          op:
+                            type: string
+                          path:
+                            type: string
+                          value:
+                            type: object
+                        required:
+                        - path
+                        type: object
+                      type: array
+                  required:
+                  - clusterName
+                  type: object
+                type: array
+              placement:
+                properties:
+                  clusterSelector:
+                    description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  clusters:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                      required:
+                      - name
+                      type: object
+                    type: array
+                type: object
+              template:
+                properties:
+                  metadata:
+                    type: object
+                  spec:
+                    description: WorkspaceSpec defines the desired state of Workspace
+                    properties:
+                      manager:
+                        type: string
+                      networkIsolation:
+                        type: boolean
+                    type: object
+                type: object
+            required:
+            - placement
+            - template
+            type: object
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/tenant_v1alpha1_workspace.yaml

@@ -1,42 +0,0 @@
-apiVersion: apiextensions.k8s.io/v1beta1
-kind: CustomResourceDefinition
-metadata:
-  creationTimestamp: null
-  labels:
-    controller-tools.k8s.io: "1.0"
-  name: workspaces.tenant.kubesphere.io
-spec:
-  group: tenant.kubesphere.io
-  names:
-    kind: Workspace
-    plural: workspaces
-  scope: Cluster
-  validation:
-    openAPIV3Schema:
-      properties:
-        apiVersion:
-          description: 'APIVersion defines the versioned schema of this representation
-            of an object. Servers should convert recognized schemas to the latest
-            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
-          type: string
-        kind:
-          description: 'Kind is a string value representing the REST resource this
-            object represents. Servers may infer this from the endpoint the client
-            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
-          type: string
-        metadata:
-          type: object
-        spec:
-          properties:
-            manager:
-              type: string
-          type: object
-        status:
-          type: object
-  version: v1alpha1
-status:
-  acceptedNames:
-    kind: ""
-    plural: ""
-  conditions: []
-  storedVersions: []

config/crds/types.kubefed.io_federatedapplications.yaml

@@ -0,0 +1,440 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: federatedapplications.types.kubefed.io
+spec:
+  group: types.kubefed.io
+  names:
+    kind: FederatedApplication
+    listKind: FederatedApplicationList
+    plural: federatedapplications
+    singular: federatedapplication
+  scope: Namespaced
+  versions:
+  - name: v1beta1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              overrides:
+                items:
+                  properties:
+                    clusterName:
+                      type: string
+                    clusterOverrides:
+                      items:
+                        properties:
+                          op:
+                            type: string
+                          path:
+                            type: string
+                          value:
+                            type: object
+                        required:
+                        - path
+                        type: object
+                      type: array
+                  required:
+                  - clusterName
+                  type: object
+                type: array
+              placement:
+                properties:
+                  clusterSelector:
+                    description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  clusters:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                      required:
+                      - name
+                      type: object
+                    type: array
+                type: object
+              template:
+                properties:
+                  spec:
+                    description: ApplicationSpec defines the specification for an Application.
+                    properties:
+                      addOwnerRef:
+                        description: AddOwnerRef objects - flag to indicate if we need to add OwnerRefs to matching objects Matching is done by using Selector to query all ComponentGroupKinds
+                        type: boolean
+                      assemblyPhase:
+                        description: AssemblyPhase represents the current phase of the application's assembly. An empty value is equivalent to "Succeeded".
+                        type: string
+                      componentKinds:
+                        description: ComponentGroupKinds is a list of Kinds for Application's components (e.g. Deployments, Pods, Services, CRDs). It can be used in conjunction with the Application's Selector to list or watch the Applications components.
+                        items:
+                          description: GroupKind specifies a Group and a Kind, but does not force a version.  This is useful for identifying concepts during lookup stages without having partially valid types
+                          properties:
+                            group:
+                              type: string
+                            kind:
+                              type: string
+                          required:
+                          - group
+                          - kind
+                          type: object
+                        type: array
+                      descriptor:
+                        description: Descriptor regroups information and metadata about an application.
+                        properties:
+                          description:
+                            description: Description is a brief string description of the Application.
+                            type: string
+                          icons:
+                            description: Icons is an optional list of icons for an application. Icon information includes the source, size, and mime type.
+                            items:
+                              description: ImageSpec contains information about an image used as an icon.
+                              properties:
+                                size:
+                                  description: (optional) The size of the image in pixels (e.g., 25x25).
+                                  type: string
+                                src:
+                                  description: The source for image represented as either an absolute URL to the image or a Data URL containing the image. Data URLs are defined in RFC 2397.
+                                  type: string
+                                type:
+                                  description: (optional) The mine type of the image (e.g., "image/png").
+                                  type: string
+                              required:
+                              - src
+                              type: object
+                            type: array
+                          keywords:
+                            description: Keywords is an optional list of key words associated with the application (e.g. MySQL, RDBMS, database).
+                            items:
+                              type: string
+                            type: array
+                          links:
+                            description: Links are a list of descriptive URLs intended to be used to surface additional documentation, dashboards, etc.
+                            items:
+                              description: Link contains information about an URL to surface documentation, dashboards, etc.
+                              properties:
+                                description:
+                                  description: Description is human readable content explaining the purpose of the link.
+                                  type: string
+                                url:
+                                  description: Url typically points at a website address.
+                                  type: string
+                              type: object
+                            type: array
+                          maintainers:
+                            description: Maintainers is an optional list of maintainers of the application. The maintainers in this list maintain the the source code, images, and package for the application.
+                            items:
+                              description: ContactData contains information about an individual or organization.
+                              properties:
+                                email:
+                                  description: Email is the email address.
+                                  type: string
+                                name:
+                                  description: Name is the descriptive name.
+                                  type: string
+                                url:
+                                  description: Url could typically be a website address.
+                                  type: string
+                              type: object
+                            type: array
+                          notes:
+                            description: Notes contain a human readable snippets intended as a quick start for the users of the Application. CommonMark markdown syntax may be used for rich text representation.
+                            type: string
+                          owners:
+                            description: Owners is an optional list of the owners of the installed application. The owners of the application should be contacted in the event of a planned or unplanned disruption affecting the application.
+                            items:
+                              description: ContactData contains information about an individual or organization.
+                              properties:
+                                email:
+                                  description: Email is the email address.
+                                  type: string
+                                name:
+                                  description: Name is the descriptive name.
+                                  type: string
+                                url:
+                                  description: Url could typically be a website address.
+                                  type: string
+                              type: object
+                            type: array
+                          type:
+                            description: Type is the type of the application (e.g. WordPress, MySQL, Cassandra).
+                            type: string
+                          version:
+                            description: Version is an optional version indicator for the Application.
+                            type: string
+                        type: object
+                      info:
+                        description: Info contains human readable key,value pairs for the Application.
+                        items:
+                          description: InfoItem is a human readable key,value pair containing important information about how to access the Application.
+                          properties:
+                            name:
+                              description: Name is a human readable title for this piece of information.
+                              type: string
+                            type:
+                              description: Type of the value for this InfoItem.
+                              type: string
+                            value:
+                              description: Value is human readable content.
+                              type: string
+                            valueFrom:
+                              description: ValueFrom defines a reference to derive the value from another source.
+                              properties:
+                                configMapKeyRef:
+                                  description: Selects a key of a ConfigMap.
+                                  properties:
+                                    apiVersion:
+                                      description: API version of the referent.
+                                      type: string
+                                    fieldPath:
+                                      description: 'If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: "spec.containers{name}" (where "name" refers to the name of the container that triggered the event) or if no container name is specified "spec.containers[2]" (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object. TODO: this design is not final and this field is subject to change in the future.'
+                                      type: string
+                                    key:
+                                      description: The key to select.
+                                      type: string
+                                    kind:
+                                      description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                                      type: string
+                                    name:
+                                      description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                                      type: string
+                                    namespace:
+                                      description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                                      type: string
+                                    resourceVersion:
+                                      description: 'Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                                      type: string
+                                    uid:
+                                      description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                                      type: string
+                                  type: object
+                                ingressRef:
+                                  description: Select an Ingress.
+                                  properties:
+                                    apiVersion:
+                                      description: API version of the referent.
+                                      type: string
+                                    fieldPath:
+                                      description: 'If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: "spec.containers{name}" (where "name" refers to the name of the container that triggered the event) or if no container name is specified "spec.containers[2]" (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object. TODO: this design is not final and this field is subject to change in the future.'
+                                      type: string
+                                    host:
+                                      description: The optional host to select.
+                                      type: string
+                                    kind:
+                                      description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                                      type: string
+                                    name:
+                                      description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                                      type: string
+                                    namespace:
+                                      description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                                      type: string
+                                    path:
+                                      description: The optional HTTP path.
+                                      type: string
+                                    protocol:
+                                      description: Protocol for the ingress
+                                      type: string
+                                    resourceVersion:
+                                      description: 'Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                                      type: string
+                                    uid:
+                                      description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                                      type: string
+                                  type: object
+                                secretKeyRef:
+                                  description: Selects a key of a Secret.
+                                  properties:
+                                    apiVersion:
+                                      description: API version of the referent.
+                                      type: string
+                                    fieldPath:
+                                      description: 'If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: "spec.containers{name}" (where "name" refers to the name of the container that triggered the event) or if no container name is specified "spec.containers[2]" (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object. TODO: this design is not final and this field is subject to change in the future.'
+                                      type: string
+                                    key:
+                                      description: The key to select.
+                                      type: string
+                                    kind:
+                                      description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                                      type: string
+                                    name:
+                                      description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                                      type: string
+                                    namespace:
+                                      description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                                      type: string
+                                    resourceVersion:
+                                      description: 'Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                                      type: string
+                                    uid:
+                                      description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                                      type: string
+                                  type: object
+                                serviceRef:
+                                  description: Select a Service.
+                                  properties:
+                                    apiVersion:
+                                      description: API version of the referent.
+                                      type: string
+                                    fieldPath:
+                                      description: 'If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: "spec.containers{name}" (where "name" refers to the name of the container that triggered the event) or if no container name is specified "spec.containers[2]" (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object. TODO: this design is not final and this field is subject to change in the future.'
+                                      type: string
+                                    kind:
+                                      description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+                                      type: string
+                                    name:
+                                      description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                                      type: string
+                                    namespace:
+                                      description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                                      type: string
+                                    path:
+                                      description: The optional HTTP path.
+                                      type: string
+                                    port:
+                                      description: The optional port to select.
+                                      format: int32
+                                      type: integer
+                                    protocol:
+                                      description: Protocol for the service
+                                      type: string
+                                    resourceVersion:
+                                      description: 'Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'
+                                      type: string
+                                    uid:
+                                      description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                                      type: string
+                                  type: object
+                                type:
+                                  description: Type of source.
+                                  type: string
+                              type: object
+                          type: object
+                        type: array
+                      selector:
+                        description: 'Selector is a label query over kinds that created by the application. It must match the component objects'' labels. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors'
+                        properties:
+                          matchExpressions:
+                            description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                            items:
+                              description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                              properties:
+                                key:
+                                  description: key is the label key that the selector applies to.
+                                  type: string
+                                operator:
+                                  description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                                  type: string
+                                values:
+                                  description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                                  items:
+                                    type: string
+                                  type: array
+                              required:
+                              - key
+                              - operator
+                              type: object
+                            type: array
+                          matchLabels:
+                            additionalProperties:
+                              type: string
+                            description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                            type: object
+                        type: object
+                    type: object
+                type: object
+            required:
+            - placement
+            - template
+            type: object
+          status:
+            properties:
+              clusters:
+                items:
+                  properties:
+                    name:
+                      type: string
+                    status:
+                      type: string
+                  required:
+                  - name
+                  type: object
+                type: array
+              conditions:
+                items:
+                  properties:
+                    lastTransitionTime:
+                      description: Last time the condition transit from one status to another.
+                      type: string
+                    lastUpdateTime:
+                      description: Last time reconciliation resulted in an error or the last time a change was propagated to member clusters.
+                      type: string
+                    reason:
+                      description: (brief) reason for the condition's last transition.
+                      type: string
+                    status:
+                      description: Status of the condition, one of True, False, Unknown.
+                      type: string
+                    type:
+                      description: Type of cluster condition
+                      type: string
+                  required:
+                  - status
+                  - type
+                  type: object
+                type: array
+              observedGeneration:
+                format: int64
+                type: integer
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/types.kubefed.io_federatedclusterrolebindings.yaml

@@ -0,0 +1,194 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: federatedclusterrolebindings.types.kubefed.io
+spec:
+  group: types.kubefed.io
+  names:
+    kind: FederatedClusterRoleBinding
+    listKind: FederatedClusterRoleBindingList
+    plural: federatedclusterrolebindings
+    singular: federatedclusterrolebinding
+  scope: Namespaced
+  versions:
+  - name: v1beta1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              overrides:
+                items:
+                  properties:
+                    clusterName:
+                      type: string
+                    clusterOverrides:
+                      items:
+                        properties:
+                          op:
+                            type: string
+                          path:
+                            type: string
+                          value:
+                            type: object
+                        required:
+                        - path
+                        type: object
+                      type: array
+                  required:
+                  - clusterName
+                  type: object
+                type: array
+              placement:
+                properties:
+                  clusterSelector:
+                    description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  clusters:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                      required:
+                      - name
+                      type: object
+                    type: array
+                type: object
+              template:
+                properties:
+                  roleRef:
+                    description: RoleRef can only reference a ClusterRole in the global namespace. If the RoleRef cannot be resolved, the Authorizer must return an error.
+                    properties:
+                      apiGroup:
+                        description: APIGroup is the group for the resource being referenced
+                        type: string
+                      kind:
+                        description: Kind is the type of resource being referenced
+                        type: string
+                      name:
+                        description: Name is the name of resource being referenced
+                        type: string
+                    required:
+                    - apiGroup
+                    - kind
+                    - name
+                    type: object
+                  subjects:
+                    items:
+                      description: Subject contains a reference to the object or user identities a role binding applies to.  This can either hold a direct API object reference, or a value for non-objects such as user and group names.
+                      properties:
+                        apiGroup:
+                          description: APIGroup holds the API group of the referenced subject. Defaults to "" for ServiceAccount subjects. Defaults to "rbac.authorization.k8s.io" for User and Group subjects.
+                          type: string
+                        kind:
+                          description: Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount". If the Authorizer does not recognized the kind value, the Authorizer should report an error.
+                          type: string
+                        name:
+                          description: Name of the object being referenced.
+                          type: string
+                        namespace:
+                          description: Namespace of the referenced object.  If the object kind is non-namespace, such as "User" or "Group", and this value is not empty the Authorizer should report an error.
+                          type: string
+                      required:
+                      - kind
+                      - name
+                      type: object
+                    type: array
+                required:
+                - roleRef
+                type: object
+            required:
+            - placement
+            - template
+            type: object
+          status:
+            properties:
+              clusters:
+                items:
+                  properties:
+                    name:
+                      type: string
+                    status:
+                      type: string
+                  required:
+                  - name
+                  type: object
+                type: array
+              conditions:
+                items:
+                  properties:
+                    lastTransitionTime:
+                      description: Last time the condition transit from one status to another.
+                      type: string
+                    lastUpdateTime:
+                      description: Last time reconciliation resulted in an error or the last time a change was propagated to member clusters.
+                      type: string
+                    reason:
+                      description: (brief) reason for the condition's last transition.
+                      type: string
+                    status:
+                      description: Status of the condition, one of True, False, Unknown.
+                      type: string
+                    type:
+                      description: Type of cluster condition
+                      type: string
+                  required:
+                  - status
+                  - type
+                  type: object
+                type: array
+              observedGeneration:
+                format: int64
+                type: integer
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/types.kubefed.io_federatedclusterroles.yaml

@@ -0,0 +1,224 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: federatedclusterroles.types.kubefed.io
+spec:
+  group: types.kubefed.io
+  names:
+    kind: FederatedClusterRole
+    listKind: FederatedClusterRoleList
+    plural: federatedclusterroles
+    singular: federatedclusterrole
+  scope: Namespaced
+  versions:
+  - name: v1beta1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              overrides:
+                items:
+                  properties:
+                    clusterName:
+                      type: string
+                    clusterOverrides:
+                      items:
+                        properties:
+                          op:
+                            type: string
+                          path:
+                            type: string
+                          value:
+                            type: object
+                        required:
+                        - path
+                        type: object
+                      type: array
+                  required:
+                  - clusterName
+                  type: object
+                type: array
+              placement:
+                properties:
+                  clusterSelector:
+                    description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  clusters:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                      required:
+                      - name
+                      type: object
+                    type: array
+                type: object
+              template:
+                properties:
+                  aggregationRule:
+                    description: AggregationRule describes how to locate ClusterRoles to aggregate into the ClusterRole
+                    properties:
+                      clusterRoleSelectors:
+                        description: ClusterRoleSelectors holds a list of selectors which will be used to find ClusterRoles and create the rules. If any of the selectors match, then the ClusterRole's permissions will be added
+                        items:
+                          description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
+                          properties:
+                            matchExpressions:
+                              description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                              items:
+                                description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                                properties:
+                                  key:
+                                    description: key is the label key that the selector applies to.
+                                    type: string
+                                  operator:
+                                    description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                                    type: string
+                                  values:
+                                    description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                                    items:
+                                      type: string
+                                    type: array
+                                required:
+                                - key
+                                - operator
+                                type: object
+                              type: array
+                            matchLabels:
+                              additionalProperties:
+                                type: string
+                              description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                              type: object
+                          type: object
+                        type: array
+                    type: object
+                  rules:
+                    items:
+                      description: PolicyRule holds information that describes a policy rule, but does not contain information about who the rule applies to or which namespace the rule applies to.
+                      properties:
+                        apiGroups:
+                          description: APIGroups is the name of the APIGroup that contains the resources.  If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed.
+                          items:
+                            type: string
+                          type: array
+                        nonResourceURLs:
+                          description: NonResourceURLs is a set of partial urls that a user should have access to.  *s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"),  but not both.
+                          items:
+                            type: string
+                          type: array
+                        resourceNames:
+                          description: ResourceNames is an optional white list of names that the rule applies to.  An empty set means that everything is allowed.
+                          items:
+                            type: string
+                          type: array
+                        resources:
+                          description: Resources is a list of resources this rule applies to.  ResourceAll represents all resources.
+                          items:
+                            type: string
+                          type: array
+                        verbs:
+                          description: Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule.  VerbAll represents all kinds.
+                          items:
+                            type: string
+                          type: array
+                      required:
+                      - verbs
+                      type: object
+                    type: array
+                type: object
+            required:
+            - placement
+            - template
+            type: object
+          status:
+            properties:
+              clusters:
+                items:
+                  properties:
+                    name:
+                      type: string
+                    status:
+                      type: string
+                  required:
+                  - name
+                  type: object
+                type: array
+              conditions:
+                items:
+                  properties:
+                    lastTransitionTime:
+                      description: Last time the condition transit from one status to another.
+                      type: string
+                    lastUpdateTime:
+                      description: Last time reconciliation resulted in an error or the last time a change was propagated to member clusters.
+                      type: string
+                    reason:
+                      description: (brief) reason for the condition's last transition.
+                      type: string
+                    status:
+                      description: Status of the condition, one of True, False, Unknown.
+                      type: string
+                    type:
+                      description: Type of cluster condition
+                      type: string
+                  required:
+                  - status
+                  - type
+                  type: object
+                type: array
+              observedGeneration:
+                format: int64
+                type: integer
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/types.kubefed.io_federatedconfigmaps.yaml

@@ -0,0 +1,163 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: federatedconfigmaps.types.kubefed.io
+spec:
+  group: types.kubefed.io
+  names:
+    kind: FederatedConfigMap
+    listKind: FederatedConfigMapList
+    plural: federatedconfigmaps
+    singular: federatedconfigmap
+  scope: Namespaced
+  versions:
+  - name: v1beta1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              overrides:
+                items:
+                  properties:
+                    clusterName:
+                      type: string
+                    clusterOverrides:
+                      items:
+                        properties:
+                          op:
+                            type: string
+                          path:
+                            type: string
+                          value:
+                            type: object
+                        required:
+                        - path
+                        type: object
+                      type: array
+                  required:
+                  - clusterName
+                  type: object
+                type: array
+              placement:
+                properties:
+                  clusterSelector:
+                    description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  clusters:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                      required:
+                      - name
+                      type: object
+                    type: array
+                type: object
+              template:
+                properties:
+                  binaryData:
+                    additionalProperties:
+                      format: byte
+                      type: string
+                    type: object
+                  data:
+                    additionalProperties:
+                      type: string
+                    type: object
+                type: object
+            required:
+            - placement
+            - template
+            type: object
+          status:
+            properties:
+              clusters:
+                items:
+                  properties:
+                    name:
+                      type: string
+                    status:
+                      type: string
+                  required:
+                  - name
+                  type: object
+                type: array
+              conditions:
+                items:
+                  properties:
+                    lastTransitionTime:
+                      description: Last time the condition transit from one status to another.
+                      type: string
+                    lastUpdateTime:
+                      description: Last time reconciliation resulted in an error or the last time a change was propagated to member clusters.
+                      type: string
+                    reason:
+                      description: (brief) reason for the condition's last transition.
+                      type: string
+                    status:
+                      description: Status of the condition, one of True, False, Unknown.
+                      type: string
+                    type:
+                      description: Type of cluster condition
+                      type: string
+                  required:
+                  - status
+                  - type
+                  type: object
+                type: array
+              observedGeneration:
+                format: int64
+                type: integer
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

config/crds/types.kubefed.io_federatedgroupbindings.yaml

@@ -0,0 +1,170 @@
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+  creationTimestamp: null
+  name: federatedgroupbindings.types.kubefed.io
+spec:
+  group: types.kubefed.io
+  names:
+    kind: FederatedGroupBinding
+    listKind: FederatedGroupBindingList
+    plural: federatedgroupbindings
+    singular: federatedgroupbinding
+  scope: Namespaced
+  versions:
+  - name: v1beta1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              overrides:
+                items:
+                  properties:
+                    clusterName:
+                      type: string
+                    clusterOverrides:
+                      items:
+                        properties:
+                          op:
+                            type: string
+                          path:
+                            type: string
+                          value:
+                            type: object
+                        required:
+                        - path
+                        type: object
+                      type: array
+                  required:
+                  - clusterName
+                  type: object
+                type: array
+              placement:
+                properties:
+                  clusterSelector:
+                    description: A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
+                    properties:
+                      matchExpressions:
+                        description: matchExpressions is a list of label selector requirements. The requirements are ANDed.
+                        items:
+                          description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
+                          properties:
+                            key:
+                              description: key is the label key that the selector applies to.
+                              type: string
+                            operator:
+                              description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
+                              type: string
+                            values:
+                              description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
+                              items:
+                                type: string
+                              type: array
+                          required:
+                          - key
+                          - operator
+                          type: object
+                        type: array
+                      matchLabels:
+                        additionalProperties:
+                          type: string
+                        description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
+                        type: object
+                    type: object
+                  clusters:
+                    items:
+                      properties:
+                        name:
+                          type: string
+                      required:
+                      - name
+                      type: object
+                    type: array
+                type: object
+              template:
+                properties:
+                  groupRef:
+                    description: GroupRef defines the desired relation of GroupBinding
+                    properties:
+                      apiGroup:
+                        type: string
+                      kind:
+                        type: string
+                      name:
+                        type: string
+                    type: object
+                  metadata:
+                    type: object
+                  users:
+                    items:
+                      type: string
+                    type: array
+                type: object
+            required:
+            - placement
+            - template
+            type: object
+          status:
+            properties:
+              clusters:
+                items:
+                  properties:
+                    name:
+                      type: string
+                    status:
+                      type: string
+                  required:
+                  - name
+                  type: object
+                type: array
+              conditions:
+                items:
+                  properties:
+                    lastTransitionTime:
+                      description: Last time the condition transit from one status to another.
+                      type: string
+                    lastUpdateTime:
+                      description: Last time reconciliation resulted in an error or the last time a change was propagated to member clusters.
+                      type: string
+                    reason:
+                      description: (brief) reason for the condition's last transition.
+                      type: string
+                    status:
+                      description: Status of the condition, one of True, False, Unknown.
+                      type: string
+                    type:
+                      description: Type of cluster condition
+                      type: string
+                  required:
+                  - status
+                  - type
+                  type: object
+                type: array
+              observedGeneration:
+                format: int64
+                type: integer
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []