kubernetes源码分析之kube-apiserver

最新推荐文章于 2025-02-13 15:23:36 发布
最新推荐文章于 2025-02-13 15:23:36 发布
阅读量393 收藏
点赞数
分类专栏： kubernetes GO
原文链接：https://juejin.im/post/5c934e5a5188252d7c216981
版权
本文详细解析了Kubernetes API Server的启动流程，从命令行参数配置到最终的运行状态，涵盖了参数配置、合法性验证、创建及启动server等关键步骤。深入探讨了CreateServerChain方法中apiserver配置、扩展APIserver配置与创建、聚合server配置的过程，以及CreateKubeAPIServer方法中RESTStorage的初始化与路由注册机制。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

本节所有的代码基于最新的1.15.2

启动分析

同Kubernetes所有的组件启动代码一致，apiserver启动使用的是corba的命令行方式

		RunE: func(cmd *cobra.Command, args []string) error {
			verflag.PrintAndExitIfRequested()
			utilflag.PrintFlags(cmd.Flags())

			// set default options
			// 完成参数的配置
			completedOptions, err := Complete(s)
			if err != nil {
				return err
			}

			// validate options
			// 判断配置是否合法
			if errs := completedOptions.Validate(); len(errs) != 0 {
				return utilerrors.NewAggregate(errs)
			}

			return Run(completedOptions, genericapiserver.SetupSignalHandler())
		},

启动完成需要三个步骤：

  1. 完成参数的配置
  2. 判断配置是否合法
  3. 执行最终的Run方法
func Run(completeOptions completedServerRunOptions, stopCh <-chan struct{}) error {
	// To help debugging, immediately log version
	klog.Infof("Version: %+v", version.Get())
	
    //创建server
	server, err := CreateServerChain(completeOptions, stopCh)
	if err != nil {
		return err
	}

    // 启动server
	return server.PrepareRun().Run(stopCh)
}

Run 方法主要执行两个步骤：

  1. 创建server端
  2. 启动server

apiserver本质上就是一个server服务器，所有代码核心就是如何配置server，包括路由、访问权限以及同数据库（etcd）的交互。

server端创建

server端创建主要几种在 CreateServerChain方法，方法代码如下

func CreateServerChain(completedOptions completedServerRunOptions, stopCh <-chan struct{}) (*genericapiserver.GenericAPIServer, error) {
	nodeTunneler, proxyTransport, err := CreateNodeDialer(completedOptions)
	if err != nil {
		return nil, err
	}

    //创建kubeApiserverConfig配置
	kubeAPIServerConfig, insecureServingInfo, serviceResolver, pluginInitializer, admissionPostStartHook, err := CreateKubeAPIServerConfig(completedOptions, nodeTunneler, proxyTransport)
	if err != nil {
		return nil, err
	}

	// If additional API servers are added, they should be gated.
	// 判断是否配置了扩展API server，创建apiExtensionConfig配置
	apiExtensionsConfig, err := createAPIExtensionsConfig(*kubeAPIServerConfig.GenericConfig, kubeAPIServerConfig.ExtraConfig.VersionedInformers, pluginInitializer, completedOptions.ServerRunOptions, completedOptions.MasterCount,
		serviceResolver, webhook.NewDefaultAuthenticationInfoResolverWrapper(proxyTransport, kubeAPIServerConfig.GenericConfig.LoopbackClientConfig))
	if err != nil {
		return nil, err
	}
    // 启动扩展的API server
	apiExtensionsServer, err := createAPIExtensionsServer(apiExtensionsConfig, genericapiserver.NewEmptyDelegate())
	if err != nil {
		return nil, err
	}
    
    // 启动最核心的KubeAPIServer
	kubeAPIServer, err := CreateKubeAPIServer(kubeAPIServerConfig, apiExtensionsServer.GenericAPIServer, admissionPostStartHook)
	if err != nil {
		return nil, err
	}

	// otherwise go down the normal path of standing the aggregator up in front of the API server
	// this wires up openapi
	kubeAPIServer.GenericAPIServer.PrepareRun()

	// This will wire up openapi for extension api server
	apiExtensionsServer.GenericAPIServer.PrepareRun()

	// aggregator comes last in the chain
	// 聚合层的配置aggregatorConfig
	aggregatorConfig, err := createAggregatorConfig(*kubeAPIServerConfig.GenericConfig, completedOptions.ServerRunOptions, kubeAPIServerConfig.ExtraConfig.VersionedInformers, serviceResolver, proxyTransport, pluginInitializer)
	if err != nil {
		return nil, err
	}
	//aggregatorServer聚合服务器，对所有服务器访问的整合
	aggregatorServer, err := createAggregatorServer(aggregatorConfig, kubeAPIServer.GenericAPIServer, apiExtensionsServer.Informers)
	if err != nil {
		// we don't need special handling for innerStopCh because the aggregator server doesn't create any go routines
		return nil, err
	}
  
    //启动非安全端口的server
	if insecureServingInfo != nil {
		insecureHandlerChain := kubeserver.BuildInsecureHandlerChain(aggregatorServer.GenericAPIServer.UnprotectedHandler(), kubeAPIServerConfig.GenericConfig)
		if err := insecureServingInfo.Serve(insecureHandlerChain, kubeAPIServerConfig.GenericConfig.RequestTimeout, stopCh); err != nil {
			return nil, err
		}
	}
    
    //返回 GenericAPIServer，后续启动安全端口的server
	return aggregatorServer.GenericAPIServer, nil
}

创建过程主要有以下步骤：

  1. 根据配置构造apiserver的配置，调用CreateKubeAPIServerConfig；
  2. 根据配置构建扩展的apiserver的配置，调用的方法是CreateAPIExentsionsConfig；
  3. 创建server，包括扩展的apiserver和原生的apiserver，调用的方法为CreateAPIExtensionServer 和 CreateKubeAPIServer。主要是将各个handler的路由方法注册到container中去，完全遵守go-restful的设计模式，即将处理方法注册到Route中去，同一个根路径下的Route注册到webServer中去，WebService注册到Container中，Container负责分发，访问的过程为Container → WebServer → Route
  4. 聚合server的配置和创建，主要是原生的apiserver和扩展的apiserver的访问进行整合，添加后续的一些处理接口。调用的方法为createAggregatorConfig 和 createAggregatorServer
  5. 创建完成， 返回配置的server信息，以上最核心的就是apiserver的创建

创建

CreateKubeAPIServer方法如下

func CreateKubeAPIServer(kubeAPIServerConfig *master.Config, delegateAPIServer genericapiserver.DelegationTarget, admissionPostStartHook genericapiserver.PostStartHookFunc) (*master.Master, error) {
	kubeAPIServer, err := kubeAPIServerConfig.Complete().New(delegateAPIServer)
	if err != nil {
		return nil, err
	}

	kubeAPIServer.GenericAPIServer.AddPostStartHookOrDie("start-kube-apiserver-admission-initializer", admissionPostStartHook)

	return kubeAPIServer, nil
}

通过Complete 方法完成配置的最终合法化，New方法生成kubeAPIServer的配置，进入New方法。

func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*Master, error) {
	if reflect.DeepEqual(c.ExtraConfig.KubeletClientConfig, kubeletclient.KubeletClientConfig{}) {
		return nil, fmt.Errorf("Master.New() called with empty config.KubeletClientConfig")
	}

	s, err := c.GenericConfig.New("kube-apiserver", delegationTarget)
	if err != nil {
		return nil, err
	}

	if c.ExtraConfig.EnableLogsSupport {
		routes.Logs{}.Install(s.Handler.GoRestfulContainer)
	}

	m := &Master{
		GenericAPIServer: s,
	}

	// install legacy rest storage
	if c.ExtraConfig.APIResourceConfigSource.VersionEnabled(apiv1.SchemeGroupVersion) {
		legacyRESTStorageProvider := corerest.LegacyRESTStorageProvider{
			StorageFactory:              c.ExtraConfig.StorageFactory,
			ProxyTransport:              c.ExtraConfig.ProxyTransport,
			KubeletClientConfig:         c.ExtraConfig.KubeletClientConfig,
			EventTTL:                    c.ExtraConfig.EventTTL,
			ServiceIPRange:              c.ExtraConfig.ServiceIPRange,
			ServiceNodePortRange:        c.ExtraConfig.ServiceNodePortRange,
			LoopbackClientConfig:        c.GenericConfig.LoopbackClientConfig,
			ServiceAccountIssuer:        c.ExtraConfig.ServiceAccountIssuer,
			ServiceAccountMaxExpiration: c.ExtraConfig.ServiceAccountMaxExpiration,
			APIAudiences:                c.GenericConfig.Authentication.APIAudiences,
		}
		m.InstallLegacyAPI(&c, c.GenericConfig.RESTOptionsGetter, legacyRESTStorageProvider)
	}

	// The order here is preserved in discovery.
	// If resources with identical names exist in more than one of these groups (e.g. "deployments.apps"" and "deployments.extensions"),
	// the order of this list determines which group an unqualified resource name (e.g. "deployments") should prefer.
	// This priority order is used for local discovery, but it ends up aggregated in `k8s.io/kubernetes/cmd/kube-apiserver/app/aggregator.go
	// with specific priorities.
	// TODO: describe the priority all the way down in the RESTStorageProviders and plumb it back through the various discovery
	// handlers that we have.
	restStorageProviders := []RESTStorageProvider{
		auditregistrationrest.RESTStorageProvider{},
		authenticationrest.RESTStorageProvider{Authenticator: c.GenericConfig.Authentication.Authenticator, APIAudiences: c.GenericConfig.Authentication.APIAudiences},
		authorizationrest.RESTStorageProvider{Authorizer: c.GenericConfig.Authorization.Authorizer, RuleResolver: c.GenericConfig.RuleResolver},
		autoscalingrest.RESTStorageProvider{},
		batchrest.RESTStorageProvider{},
		certificatesrest.RESTStorageProvider{},
		coordinationrest.RESTStorageProvider{},
		extensionsrest.RESTStorageProvider{},
		networkingrest.RESTStorageProvider{},
		noderest.RESTStorageProvider{},
		policyrest.RESTStorageProvider{},
		rbacrest.RESTStorageProvider{Authorizer: c.GenericConfig.Authorization.Authorizer},
		schedulingrest.RESTStorageProvider{},
		settingsrest.RESTStorageProvider{},
		storagerest.RESTStorageProvider{},
		// keep apps after extensions so legacy clients resolve the extensions versions of shared resource names.
		// See https://github.com/kubernetes/kubernetes/issues/42392
		appsrest.RESTStorageProvider{},
		admissionregistrationrest.RESTStorageProvider{},
		eventsrest.RESTStorageProvider{TTL: c.ExtraConfig.EventTTL},
	}
	m.InstallAPIs(c.ExtraConfig.APIResourceConfigSource, c.GenericConfig.RESTOptionsGetter, restStorageProviders...)

	if c.ExtraConfig.Tunneler != nil {
		m.installTunneler(c.ExtraConfig.Tunneler, corev1client.NewForConfigOrDie(c.GenericConfig.LoopbackClientConfig).Nodes())
	}

	m.GenericAPIServer.AddPostStartHookOrDie("ca-registration", c.ExtraConfig.ClientCARegistrationHook.PostStartHook)

	return m, nil
}

func (c completedConfig) New(name string, delegationTarget DelegationTarget) (*GenericAPIServer, error) {
	if c.Serializer == nil {
		return nil, fmt.Errorf("Genericapiserver.New() called with config.Serializer == nil")
	}
	if c.LoopbackClientConfig == nil {
		return nil, fmt.Errorf("Genericapiserver.New() called with config.LoopbackClientConfig == nil")
	}
	if c.EquivalentResourceRegistry == nil {
		return nil, fmt.Errorf("Genericapiserver.New() called with config.EquivalentResourceRegistry == nil")
	}

	handlerChainBuilder := func(handler http.Handler) http.Handler {
		return c.BuildHandlerChainFunc(handler, c.Config)
	}
	apiServerHandler := NewAPIServerHandler(name, c.Serializer, handlerChainBuilder, delegationTarget.UnprotectedHandler())

	s := &GenericAPIServer{
		discoveryAddresses:         c.DiscoveryAddresses,
		LoopbackClientConfig:       c.LoopbackClientConfig,
		legacyAPIGroupPrefixes:     c.LegacyAPIGroupPrefixes,
		admissionControl:           c.AdmissionControl,
		Serializer:                 c.Serializer,
		AuditBackend:               c.AuditBackend,
		Authorizer:                 c.Authorization.Authorizer,
		delegationTarget:           delegationTarget,
		EquivalentResourceRegistry: c.EquivalentResourceRegistry,
		HandlerChainWaitGroup:      c.HandlerChainWaitGroup,

		minRequestTimeout: time.Duration(c.MinRequestTimeout) * time.Second,
		ShutdownTimeout:   c.RequestTimeout,

		SecureServingInfo: c.SecureServing,
		ExternalAddress:   c.ExternalAddress,

		Handler: apiServerHandler,

		listedPathProvider: apiServerHandler,

		openAPIConfig: c.OpenAPIConfig,

		postStartHooks:         map[string]postStartHookEntry{},
		preShutdownHooks:       map[string]preShutdownHookEntry{},
		disabledPostStartHooks: c.DisabledPostStartHooks,

		healthzChecks: c.HealthzChecks,

		DiscoveryGroupManager: discovery.NewRootAPIsHandler(c.DiscoveryAddresses, c.Serializer),

		enableAPIResponseCompression: c.EnableAPIResponseCompression,
		maxRequestBodyBytes:          c.MaxRequestBodyBytes,
	}

	for {
		if c.JSONPatchMaxCopyBytes <= 0 {
			break
		}
		existing := atomic.LoadInt64(&jsonpatch.AccumulatedCopySizeLimit)
		if existing > 0 && existing < c.JSONPatchMaxCopyBytes {
			break
		}
		if atomic.CompareAndSwapInt64(&jsonpatch.AccumulatedCopySizeLimit, existing, c.JSONPatchMaxCopyBytes) {
			break
		}
	}

	for k, v := range delegationTarget.PostStartHooks() {
		s.postStartHooks[k] = v
	}

	for k, v := range delegationTarget.PreShutdownHooks() {
		s.preShutdownHooks[k] = v
	}

	genericApiServerHookName := "generic-apiserver-start-informers"
	if c.SharedInformerFactory != nil && !s.isPostStartHookRegistered(genericApiServerHookName) {
		err := s.AddPostStartHook(genericApiServerHookName, func(context PostStartHookContext) error {
			c.SharedInformerFactory.Start(context.StopCh)
			return nil
		})
		if err != nil {
			return nil, err
		}
	}

	for _, delegateCheck := range delegationTarget.HealthzChecks() {
		skip := false
		for _, existingCheck := range c.HealthzChecks {
			if existingCheck.Name() == delegateCheck.Name() {
				skip = true
				break
			}
		}
		if skip {
			continue
		}

		s.healthzChecks = append(s.healthzChecks, delegateCheck)
	}

	s.listedPathProvider = routes.ListedPathProviders{s.listedPathProvider, delegationTarget}

	installAPI(s, c.Config)

	// use the UnprotectedHandler from the delegation target to ensure that we don't attempt to double authenticator, authorize,
	// or some other part of the filter chain in delegation cases.
	if delegationTarget.UnprotectedHandler() == nil && c.EnableIndex {
		s.Handler.NonGoRestfulMux.NotFoundHandler(routes.IndexLister{
			StatusCode:   http.StatusNotFound,
			PathProvider: s.listedPathProvider,
		})
	}

	return s, nil
}
```,
1. 按照go-restful的模式，调用c.GenericConfig.New方法初始化Container，即gorestfulContainer，初始化方法为NewAPIServerHandler。初始化之后，添加路由。

```go
func installAPI(s *GenericAPIServer, c *Config) {
	// 添加“/”与“/index.html”路由
	if c.EnableIndex {
		routes.Index{}.Install(s.listedPathProvider, s.Handler.NonGoRestfulMux)
	}
	// 添加debug相关路由
	if c.EnableProfiling {
		routes.Profiling{}.Install(s.Handler.NonGoRestfulMux)
		if c.EnableContentionProfiling {
			goruntime.SetBlockProfileRate(1)
		}
		// so far, only logging related endpoints are considered valid to add for these debug flags.
		routes.DebugFlags{}.Install(s.Handler.NonGoRestfulMux, "v", routes.StringFlagPutHandler(logs.GlogSetter))
	}
	// 添加“/metrices”路由
	if c.EnableMetrics {
		if c.EnableProfiling {
			routes.MetricsWithReset{}.Install(s.Handler.NonGoRestfulMux)
		} else {
			routes.DefaultMetrics{}.Install(s.Handler.NonGoRestfulMux)
		}
	}

	routes.Version{Version: c.Version}.Install(s.Handler.GoRestfulContainer)

	if c.EnableDiscovery {
		s.Handler.GoRestfulContainer.Add(s.DiscoveryGroupManager.WebService())
	}
}

该方法添加了包括/、/debug/*、/metrics、/version 几条路由，通过访问apiserver即可看到相关的信息
2. 判断是够支持logs相关的路由，如果支持，则添加/logs路由
3. 添加以api开头的路由

	if c.ExtraConfig.APIResourceConfigSource.VersionEnabled(apiv1.SchemeGroupVersion) {
		legacyRESTStorageProvider := corerest.LegacyRESTStorageProvider{
			StorageFactory:              c.ExtraConfig.StorageFactory,
			ProxyTransport:              c.ExtraConfig.ProxyTransport,
			KubeletClientConfig:         c.ExtraConfig.KubeletClientConfig,
			EventTTL:                    c.ExtraConfig.EventTTL,
			ServiceIPRange:              c.ExtraConfig.ServiceIPRange,
			ServiceNodePortRange:        c.ExtraConfig.ServiceNodePortRange,
			LoopbackClientConfig:        c.GenericConfig.LoopbackClientConfig,
			ServiceAccountIssuer:        c.ExtraConfig.ServiceAccountIssuer,
			ServiceAccountMaxExpiration: c.ExtraConfig.ServiceAccountMaxExpiration,
			APIAudiences:                c.GenericConfig.Authentication.APIAudiences,
		}
		m.InstallLegacyAPI(&c, c.GenericConfig.RESTOptionsGetter, legacyRESTStorageProvider)
	}
  1. 添加以apis开头的路由
	restStorageProviders := []RESTStorageProvider{
		auditregistrationrest.RESTStorageProvider{},
		authenticationrest.RESTStorageProvider{Authenticator: c.GenericConfig.Authentication.Authenticator, APIAudiences: c.GenericConfig.Authentication.APIAudiences},
		authorizationrest.RESTStorageProvider{Authorizer: c.GenericConfig.Authorization.Authorizer, RuleResolver: c.GenericConfig.RuleResolver},
		autoscalingrest.RESTStorageProvider{},
		batchrest.RESTStorageProvider{},
		certificatesrest.RESTStorageProvider{},
		coordinationrest.RESTStorageProvider{},
		extensionsrest.RESTStorageProvider{},
		networkingrest.RESTStorageProvider{},
		noderest.RESTStorageProvider{},
		policyrest.RESTStorageProvider{},
		rbacrest.RESTStorageProvider{Authorizer: c.GenericConfig.Authorization.Authorizer},
		schedulingrest.RESTStorageProvider{},
		settingsrest.RESTStorageProvider{},
		storagerest.RESTStorageProvider{},
		// keep apps after extensions so legacy clients resolve the extensions versions of shared resource names.
		// See https://github.com/kubernetes/kubernetes/issues/42392
		appsrest.RESTStorageProvider{},
		admissionregistrationrest.RESTStorageProvider{},
		eventsrest.RESTStorageProvider{TTL: c.ExtraConfig.EventTTL},
	}
	m.InstallAPIs(c.ExtraConfig.APIResourceConfigSource, c.GenericConfig.RESTOptionsGetter, restStorageProviders...)

可以看出apis开头的路由明显较多，应该由于kubernetes设计之初都是以api/v1开头，以后扩展的版本以apis开头命名，现在扩展的是CRD与自定义Controller的方法扩展API，不再进行api版本的扩展，基本上代码的名称都可以在实际集群中找到对应的API

路由添加

api开头的路由通过InstallLegacyAPI方法添加，如下：

func (m *Master) InstallLegacyAPI(c *completedConfig, restOptionsGetter generic.RESTOptionsGetter, legacyRESTStorageProvider corerest.LegacyRESTStorageProvider) {
	legacyRESTStorage, apiGroupInfo, err := legacyRESTStorageProvider.NewLegacyRESTStorage(restOptionsGetter)
	if err != nil {
		klog.Fatalf("Error building core storage: %v", err)
	}

	controllerName := "bootstrap-controller"
	coreClient := corev1client.NewForConfigOrDie(c.GenericConfig.LoopbackClientConfig)
	bootstrapController := c.NewBootstrapController(legacyRESTStorage, coreClient, coreClient, coreClient, coreClient.RESTClient())
	m.GenericAPIServer.AddPostStartHookOrDie(controllerName, bootstrapController.PostStartHook)
	m.GenericAPIServer.AddPreShutdownHookOrDie(controllerName, bootstrapController.PreShutdownHook)

	if err := m.GenericAPIServer.InstallLegacyAPIGroup(genericapiserver.DefaultLegacyAPIPrefix, &apiGroupInfo); err != nil {
		klog.Fatalf("Error in registering group versions: %v", err)
	}
}

通过NewLegacyRESTStorage方法创建各个资源的RESTStorage，RESTStorage是一个结构体，结构体内主要包含NewFunc返回特定资源信息、NewListFunc返回特定资源列表，CreateStrategy特定资源创建时的策略，UpdateStrategy更新时的策略以及DeleteStrategy删除时的策略等重要方法

在NewLegacyRESTStorage内部，可以看到创建多种资源的RESTStorage，常见的像event，secret，namespace，endpoints等，统一调用NewREST方法构造相应的资源，待所有资源的store创建完成之后，使用restStoreMap的Map类型将每个资源的路由和对应的strore对应起来，方便去做路由的规划

func (c LegacyRESTStorageProvider) NewLegacyRESTStorage(restOptionsGetter generic.RESTOptionsGetter) (LegacyRESTStorage, genericapiserver.APIGroupInfo, error) {
	apiGroupInfo := genericapiserver.APIGroupInfo{
		PrioritizedVersions:          legacyscheme.Scheme.PrioritizedVersionsForGroup(""),
		VersionedResourcesStorageMap: map[string]map[string]rest.Storage{},
		Scheme:                       legacyscheme.Scheme,
		ParameterCodec:               legacyscheme.ParameterCodec,
		NegotiatedSerializer:         legacyscheme.Codecs,
	}

	var podDisruptionClient policyclient.PodDisruptionBudgetsGetter
	if policyGroupVersion := (schema.GroupVersion{Group: "policy", Version: "v1beta1"}); legacyscheme.Scheme.IsVersionRegistered(policyGroupVersion) {
		var err error
		podDisruptionClient, err = policyclient.NewForConfig(c.LoopbackClientConfig)
		if err != nil {
			return LegacyRESTStorage{}, genericapiserver.APIGroupInfo{}, err
		}
	}
	restStorage := LegacyRESTStorage{}

	podTemplateStorage := podtemplatestore.NewREST(restOptionsGetter)

	eventStorage := eventstore.NewREST(restOptionsGetter, uint64(c.EventTTL.Seconds()))
	limitRangeStorage := limitrangestore.NewREST(restOptionsGetter)

	resourceQuotaStorage, resourceQuotaStatusStorage := resourcequotastore.NewREST(restOptionsGetter)
	secretStorage := secretstore.NewREST(restOptionsGetter)
	persistentVolumeStorage, persistentVolumeStatusStorage := pvstore.NewREST(restOptionsGetter)
	persistentVolumeClaimStorage, persistentVolumeClaimStatusStorage := pvcstore.NewREST(restOptionsGetter)
	configMapStorage := configmapstore.NewREST(restOptionsGetter)

	namespaceStorage, namespaceStatusStorage, namespaceFinalizeStorage := namespacestore.NewREST(restOptionsGetter)

	endpointsStorage := endpointsstore.NewREST(restOptionsGetter)

	nodeStorage, err := nodestore.NewStorage(restOptionsGetter, c.KubeletClientConfig, c.ProxyTransport)
	if err != nil {
		return LegacyRESTStorage{}, genericapiserver.APIGroupInfo{}, err
	}

	podStorage := podstore.NewStorage(
		restOptionsGetter,
		nodeStorage.KubeletConnectionInfo,
		c.ProxyTransport,
		podDisruptionClient,
	)

	var serviceAccountStorage *serviceaccountstore.REST
	if c.ServiceAccountIssuer != nil && utilfeature.DefaultFeatureGate.Enabled(features.TokenRequest) {
		serviceAccountStorage = serviceaccountstore.NewREST(restOptionsGetter, c.ServiceAccountIssuer, c.APIAudiences, c.ServiceAccountMaxExpiration, podStorage.Pod.Store, secretStorage.Store)
	} else {
		serviceAccountStorage = serviceaccountstore.NewREST(restOptionsGetter, nil, nil, 0, nil, nil)
	}

	serviceRESTStorage, serviceStatusStorage := servicestore.NewGenericREST(restOptionsGetter)

	var serviceClusterIPRegistry rangeallocation.RangeRegistry
	serviceClusterIPRange := c.ServiceIPRange
	if serviceClusterIPRange.IP == nil {
		return LegacyRESTStorage{}, genericapiserver.APIGroupInfo{}, fmt.Errorf("service clusterIPRange is missing")
	}

	serviceStorageConfig, err := c.StorageFactory.NewConfig(api.Resource("services"))
	if err != nil {
		return LegacyRESTStorage{}, genericapiserver.APIGroupInfo{}, err
	}

	serviceClusterIPAllocator := ipallocator.NewAllocatorCIDRRange(&serviceClusterIPRange, func(max int, rangeSpec string) allocator.Interface {
		mem := allocator.NewAllocationMap(max, rangeSpec)
		// TODO etcdallocator package to return a storage interface via the storageFactory
		etcd := serviceallocator.NewEtcd(mem, "/ranges/serviceips", api.Resource("serviceipallocations"), serviceStorageConfig)
		serviceClusterIPRegistry = etcd
		return etcd
	})
	restStorage.ServiceClusterIPAllocator = serviceClusterIPRegistry

	var serviceNodePortRegistry rangeallocation.RangeRegistry
	serviceNodePortAllocator := portallocator.NewPortAllocatorCustom(c.ServiceNodePortRange, func(max int, rangeSpec string) allocator.Interface {
		mem := allocator.NewAllocationMap(max, rangeSpec)
		// TODO etcdallocator package to return a storage interface via the storageFactory
		etcd := serviceallocator.NewEtcd(mem, "/ranges/servicenodeports", api.Resource("servicenodeportallocations"), serviceStorageConfig)
		serviceNodePortRegistry = etcd
		return etcd
	})
	restStorage.ServiceNodePortAllocator = serviceNodePortRegistry

	controllerStorage := controllerstore.NewStorage(restOptionsGetter)

	serviceRest, serviceRestProxy := servicestore.NewREST(serviceRESTStorage, endpointsStorage, podStorage.Pod, serviceClusterIPAllocator, serviceNodePortAllocator, c.ProxyTransport)

	restStorageMap := map[string]rest.Storage{
		"pods":             podStorage.Pod,
		"pods/attach":      podStorage.Attach,
		"pods/status":      podStorage.Status,
		"pods/log":         podStorage.Log,
		"pods/exec":        podStorage.Exec,
		"pods/portforward": podStorage.PortForward,
		"pods/proxy":       podStorage.Proxy,
		"pods/binding":     podStorage.Binding,
		"bindings":         podStorage.Binding,

		"podTemplates": podTemplateStorage,

		"replicationControllers":        controllerStorage.Controller,
		"replicationControllers/status": controllerStorage.Status,

		"services":        serviceRest,
		"services/proxy":  serviceRestProxy,
		"services/status": serviceStatusStorage,

		"endpoints": endpointsStorage,

		"nodes":        nodeStorage.Node,
		"nodes/status": nodeStorage.Status,
		"nodes/proxy":  nodeStorage.Proxy,

		"events": eventStorage,

		"limitRanges":                   limitRangeStorage,
		"resourceQuotas":                resourceQuotaStorage,
		"resourceQuotas/status":         resourceQuotaStatusStorage,
		"namespaces":                    namespaceStorage,
		"namespaces/status":             namespaceStatusStorage,
		"namespaces/finalize":           namespaceFinalizeStorage,
		"secrets":                       secretStorage,
		"serviceAccounts":               serviceAccountStorage,
		"persistentVolumes":             persistentVolumeStorage,
		"persistentVolumes/status":      persistentVolumeStatusStorage,
		"persistentVolumeClaims":        persistentVolumeClaimStorage,
		"persistentVolumeClaims/status": persistentVolumeClaimStatusStorage,
		"configMaps":                    configMapStorage,

		"componentStatuses": componentstatus.NewStorage(componentStatusStorage{c.StorageFactory}.serversToValidate),
	}
	if legacyscheme.Scheme.IsVersionRegistered(schema.GroupVersion{Group: "autoscaling", Version: "v1"}) {
		restStorageMap["replicationControllers/scale"] = controllerStorage.Scale
	}
	if legacyscheme.Scheme.IsVersionRegistered(schema.GroupVersion{Group: "policy", Version: "v1beta1"}) {
		restStorageMap["pods/eviction"] = podStorage.Eviction
	}
	if serviceAccountStorage.Token != nil {
		restStorageMap["serviceaccounts/token"] = serviceAccountStorage.Token
	}
	apiGroupInfo.VersionedResourcesStorageMap["v1"] = restStorageMap

	return restStorage, apiGroupInfo, nil
}

最终完成以api开头所有资源的RESTStorage操作。创建完成之后，则开始进行路由的安装，执行InstallLegacyAPIGroup方法，主要调用链为InstallLegacyAPIGroup →installAPIResources→InstallREST→Install→registerResourceHandlers，最终核心的路由构造在registerResourceHandlers方法内，这是一个非常复杂的方法，整个方法的代码在700行左右，方法的主要功能是通过上一步骤构造的RESTStorage判断该资源可以执行哪些操作（如create、update等），将其对应的操作存入到action，每个action对应一个标准的rest操作，如create对应的action操作为POST、update对应的action操作为PUT、最终根据action数组依次遍历，对每一个操作添加一个handler方法，注册到route中去，route注册到webservice中去，完美匹配go-restful的设计模式。

路由添加（apis开头）

api开头的路由主要是对基础资源路由的实现，而对于其他附加的资源，如：认证、网络相关的等各种扩展的api资源，统一以apis开头命名，实现入口为installAPIs,InstallAPIs与InstallLegacyAPIGroup主要区别是获取RESTStorage的方式，对于api开头的路由来说，都是api/v1这种统一的格式；而对于apis开头路由则不一样，它包含了多种不同的格式（kubernetes代码内叫groupName），如/apis/apps、/apis/certificates.k8s.io等各种无规律的groupName，为此，kubernets实现了一种RESTStorageProvider的工厂模式的接口

func (m *Master) InstallAPIs(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter, restStorageProviders ...RESTStorageProvider) {
	apiGroupsInfo := []*genericapiserver.APIGroupInfo{}

	for _, restStorageBuilder := range restStorageProviders {
		groupName := restStorageBuilder.GroupName()
		if !apiResourceConfigSource.AnyVersionForGroupEnabled(groupName) {
			klog.V(1).Infof("Skipping disabled API group %q.", groupName)
			continue
		}
		apiGroupInfo, enabled := restStorageBuilder.NewRESTStorage(apiResourceConfigSource, restOptionsGetter)
		if !enabled {
			klog.Warningf("Problem initializing API group %q, skipping.", groupName)
			continue
		}
		klog.V(1).Infof("Enabling API group %q.", groupName)

		if postHookProvider, ok := restStorageBuilder.(genericapiserver.PostStartHookProvider); ok {
			name, hook, err := postHookProvider.PostStartHook()
			if err != nil {
				klog.Fatalf("Error building PostStartHook: %v", err)
			}
			m.GenericAPIServer.AddPostStartHookOrDie(name, hook)
		}

		apiGroupsInfo = append(apiGroupsInfo, &apiGroupInfo)
	}

	if err := m.GenericAPIServer.InstallAPIGroups(apiGroupsInfo...); err != nil {
		klog.Fatalf("Error in registering group versions: %v", err)
	}
}

type RESTStorageProvider interface {
	GroupName() string
	NewRESTStorage(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter) (genericapiserver.APIGroupInfo, bool)
}

所有以apis开头的路由的资源都需要实现该接口，GroupName()方法获取到的就是类似于/apis/apps、/apis/certificates.k8s.io这样的groupName，NewRESTStorage方法获取到的是相对应的RESTStorage封装后的信息

Server端启动
通过CreateServerChain创建完server后，继续调用GenericAPIServer的Run方法完成最终的启动工作，首先通过PrepareRun方法完成启动前的路由收尾工作，该方法主要完成Swagger和OpenAPI路由的注册工作，（Swagger和OpenAPI主要包含kubernetes API的所有细节与规范）并完成/healthz路由的注册工作，完成后，开始Server启动工作。Run方法通过NonBlockingRun方法启动http server（非安全端口的启动在CreateServerChain方法已经完成）

// Run runs the specified APIServer.  This should never exit.
func Run(completeOptions completedServerRunOptions, stopCh <-chan struct{}) error {
	// To help debugging, immediately log version
	klog.Infof("Version: %+v", version.Get())

	server, err := CreateServerChain(completeOptions, stopCh)
	if err != nil {
		return err
	}

	return server.PrepareRun().Run(stopCh)
}

func (s preparedGenericAPIServer) Run(stopCh <-chan struct{}) error {
	err := s.NonBlockingRun(stopCh)
	if err != nil {
		return err
	}

	<-stopCh

	err = s.RunPreShutdownHooks()
	if err != nil {
		return err
	}

	// Wait for all requests to finish, which are bounded by the RequestTimeout variable.
	s.HandlerChainWaitGroup.Wait()

	return nil
}
【k8s源码分析-Apiserver-2】kube-apiserver 结构概览以及主体部分源码分析
叮当猫的喵i
12-25 1549
假 设 所 有 的 认 证 器 都 被 启 用 ， 当 客 户 端 发 送 请 求 到 kubeapiserver服务，该请求会进入Authentication Handler函数（处理认 证相关的Handler函数），在Authentication Handler函数中，会遍历已启用的认证器列表， 尝试执行每个认证器， 当有一个认证器返回 true时，则认证成功，否则继续尝试下一个认证器。当客户端发起一个请求，经过认证阶段时，只要有一个认证器通过，则认证成功。在客户端请求通过认证之后， 会来到授权阶段。
kube-apiserver源码分析
fengcai_ke的博客
09-12 844
kube-apiserver流程分析
kubernetes的kube-apiserver组件源码分析
虾米的博客
08-24 1826
1 apiserver模块main函数 func main(){    runtime.GOMAXPROCS(runtime.NumCPU())//多核cpu，增加系统吞吐量    rand.Seed(time.Now().UTC().UnixNano())    s:=options.NewAPIServer()//新建一个APIServer对象，APIServer结构体    s.A
Kubernetes 源码分析之 Apiserver
小太阳啊凯的博客
09-25 696
1. 概述 本文以Kubernetes 1.9 进行分析。 Kubernetes 集群中，API Server 扮演着交互入口的位置。API Server 不仅负责和 etcd 交互（其他组件不会直接操作 etcd，只有 API Server 这么做），并切对外提供统一的API调用入口, 所有的交互都是以 API Server 为核心的。API Server 提供了一下的功能： 整个集群管理的 API 接口：所有对集群进行的查询和管理都要通过 API 来进行。集群内部的组件(如kubelet)也是通过Ap
kubernetes之kube-ApiServer代码分析
weixin_33737134的博客
11-28 215
一、概述： 　　kube-ApiServer相当于是k8集群的一个入口，不论通过kubectl还是使用remote api 直接控制，都要经过apiserver。apiserver说白了就是一个server负责监听指定的端口（http/https协议），之后处理不同的请求，只不过加上的很多控制；apiserver是k8s系统中所有对象的增删查改盯的http/restful式服务端，其中盯是指wat...
kubernetes-kube-apiserver进程源码分析
hahachenchen789的博客
02-17 2340
kubernetes API server是由kube-apiserver进程实现的，它运行在kubernetes的管理节点—master上并对外提供kubernetes Restful  API服务，它提供的主要是与集群管理相关的API服务，例如校验pod、services、replication controller的配置并存储到后端的etcd server上。下面我们分别对其启动过程、关键代...
Kubernetes apiserver 源码解析
RayPick的博客
03-01 2329
要编译 Kubernetes 项目，需要先配置好 Golang 环境，本文以 v1.15.2 版本为例。Kubernetes 项目源代码可在 https://github.com/kubernetes/kubernetes 找到。
kubernetes源码分析-kube-apiserver
xiyanxiyan10的专栏
02-13 479
基于属性的访问控制（Attribute-based access control - ABAC）定义了访问控制范例，其中通过使用将属性组合在一起的策略来向用户授予访问权限。启用ABAC鉴权器需要额外增加一个。
Kubernetes源码分析-kube-proxy
最新发布
xiyanxiyan10的专栏
02-13 815
对于iptables和IPVS模式，kube-proxy的响应时间开销与建立连接相关，而不是与在这些连接上发送的数据包或请求的数量有关。这是因为Linux使用的连接跟踪（conntrack）能够非常有效地将。
kube-apiserver原理报告
qq_42944740的博客
04-03 1096
CRDAA无需编程。用户可选任何语言来实现 CRD 控制器需要编程，并构建可执行文件和镜像。无需额外运行服务；CRD 由 API 服务器处理需要额外创建服务，且该服务可能失效。一旦 CRD 被创建，不需要持续提供支持。Kubernetes 主控节点升级过程中自动会带入缺陷修复。可能需要周期性地从上游提取缺陷修复并更新聚合 API 服务器。无需处理 API 的多个版本；例如，当你控制资源的客户端时，你可以更新它使之与 API 同步。需要处理 API 的多个版本；
Kubernetes源码学习-APIServer-总览篇
ywq935的博客
12-01 361
Kubernetes源码学习-APIServer-总览篇 开胃菜-基础结构信息 APIServer的启动流程 APIServer的认证机制 APIServer的授权机制 概述 APIServer提供了 k8s各类资源对象的CURD/watch、认证授权、准入控制等众多核心功能，在k8s中定位类似于大脑和心脏，它的功能包括： 提供了集群管理的REST API接口(包括资源CURD、认证授权、数据校验以及集群状态变更)； 是所有模块的数据交互和通信的枢纽，各模块的运作都依赖于APIS
kubernetes api-server 源码阅读(二) 函数介绍
weixin_30802171的博客
03-12 219
导航： 　　1. CreateServerChain 　　　　1.1 CreateNodeDialer 　　　　1.2 CreateKubeAPIServerConfig　 　　　　1.3 createAPIExtensionsConfig 　　　　1.4createAPIExtensionsServer 　　　　1.5CreateKubeAPIServer 　...
kubernetes源码分析（1）：kube-apiserver
yjk13703623757的博客
03-03 1536
https://www.cnblogs.com/chris-cp/p/6108821.html
kubernetes apiserver源码分析二之路由
qq_21559191的专栏
04-13 544
apiserver的man函数在 k8s.io/kubernetes/cmd/kube-apiserver 目录。 但是大部分源码却在 k8s.io/apiserver cmd 目录下的kube-apiserver只是提供了一些启动参数。 所以路由代码我们要从 k8s.io/apiserver/server目录开始看 pkg目录下 admission 提供了一些权限管理的控制器 apis 是由...
编译k8s1.11 kube-apiserver源码
weixin_34357928的博客
05-26 239
说明 修改k8s1.11 kube-apiserver源码并构建成镜像，最终替换运行在集群中的kube-apiserver pod来验证本次源码修改 参看链接：https://xinchen.blog.youkuaiyun.com/article/details/88603293 centos7.4 cpu 2core mem 4g 安装go环境 yum install -y go [ro...
Kubernetes api-server源码阅读1（源码环境安装篇）
a1369760658的博客
12-20 1218
本文是 Kubernetes api-server源码阅读 系列第一篇，主要讲述如何进行kubernetes源码阅读环境的搭建，参考b站的 Kubernetes源码开发之旅 系列课程
KubeApiserver源码分析
a1023934860的博客
05-24 381
KubeApiserver流程分析 kube-apiserver底层原理使用go-restful框架对路径进行映射，然后通过etcd客户端对etcd进行请求。 我们需要注意kube-apiserver的几个功能：TLS认证、权限验证、webhook。 kube-apiserver使用http2.0作为传输协议所以对请求进行采用TLS认证，在认证完成后会通过类似于过滤链的形式对请求进行调用。 接下来让我们先看一下kube-apiserver是如何创建go-restful以及添加资源映射的。 //创建go-re
kube-apiserver鉴权源码简析
nangonghen的博客
11-19 596
kube-apiserver的鉴权其实so easy
深入分析Kubernetes源码结构
- Kubernetes的控制平面主要由kube-apiserver、kube-scheduler、kube-controller-manager和etcd组成。 - kube-apiserver是集群的前端，提供RESTful API供用户和集群的其它部分使用。 - kube-scheduler负责调度Pod...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值