kube-prometheus-0.7 部署出错问题求助 - V2EX
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
ice2016
V2EX    Kubernetes

kube-prometheus-0.7 部署出错问题求助

  •  
  •   ice2016 2021-08-31 11:27:23 +08:00 2632 次点击
    这是一个创建于 1502 天前的主题,其中的信息可能已经有所发展或是发生改变。

    hUdgY9.png 系统环境是 CentOS7.9 + K8s v1.19.14 ,根据官方兼容性说明下载安装的是 kube-prometheus-release-0.7 版本,老是报错,安装包是 https://github.com/prometheus-operator/kube-prometheus/archive/refs/heads/release-0.7.zip

    cd /root/kube-prometheus-release-0.7/manifests kubectl create -f setup 
    [root@k8s-master manifests]# kubectl get pod -o wide -n monitoring NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES prometheus-operator-7649c7454f-q9m29 1/2 CrashLoopBackOff 6 10m 10.201.1.58 k8s-node1 <none> <none> 

    查看日志:

    [root@k8s-master manifests]# kubectl logs prometheus-operator-7649c7454f-q9m29 --all-containers -n monitoring level=info ts=2021-08-31T03:16:07.410585761Z caller=main.go:235 msg="Starting Prometheus Operator" version="(version=0.44.1, branch=refs/tags/pkg/apis/monitoring/v0.44.1, revision=d8b7d3766225908d0239fd0d78258892cd0fc384)" level=info ts=2021-08-31T03:16:07.410684044Z caller=main.go:236 build_cOntext="(go=go1.14.12, user=simonpasquier, date=20201209-17:17:30)" ts=2021-08-31T03:16:07.415831751Z caller=main.go:107 msg="Starting insecure server on [::]:8080" ts=2021-08-31T03:16:10.422020132Z caller=main.go:422 msg="Unhandled error received. Exiting..." err="communicating with server failed: Get \"https://10.96.0.1:443/version?timeout=32s\": dial tcp 10.96.0.1:443: connect: connection timed out" I0831 03:10:09.496650 1 main.go:190] Valid token audiences: I0831 03:10:09.496764 1 main.go:262] Generating self signed cert as no cert is provided I0831 03:10:10.447004 1 main.go:311] Starting TCP socket on :8443 I0831 03:10:10.447347 1 main.go:318] Listening securely on :8443 

    根据报错测试几个节点上 10.96.0.1:443 这个地址都可以访问

    [root@k8s-master manifests]# curl -k https://10.96.0.1:443/version?timeout=32s { "major": "1", "minor": "19", "gitVersion": "v1.19.14", "gitCommit": "0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", "gitTreeState": "clean", "buildDate": "2021-08-11T18:02:17Z", "goVersion": "go1.15.15", "compiler": "gc", "platform": "linux/amd64" 

    iptables 信息:

    [root@k8s-master manifests]# iptables -L -n Chain INPUT (policy ACCEPT) target prot opt source destination KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */ KUBE-EXTERNAL-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */ KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0 Chain FORWARD (policy ACCEPT) target prot opt source destination KUBE-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */ DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0 DOCKER-ISOLATION-STAGE-1 all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 10.244.0.0/16 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 10.244.0.0/16 Chain OUTPUT (policy ACCEPT) target prot opt source destination KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */ KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0 Chain DOCKER (1 references) target prot opt source destination Chain DOCKER-ISOLATION-STAGE-1 (1 references) target prot opt source destination DOCKER-ISOLATION-STAGE-2 all -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 0.0.0.0/0 0.0.0.0/0 Chain DOCKER-ISOLATION-STAGE-2 (1 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 0.0.0.0/0 0.0.0.0/0 Chain DOCKER-USER (1 references) target prot opt source destination RETURN all -- 0.0.0.0/0 0.0.0.0/0 Chain KUBE-EXTERNAL-SERVICES (1 references) target prot opt source destination Chain KUBE-FIREWALL (2 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000 DROP all -- !127.0.0.0/8 127.0.0.0/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT Chain KUBE-FORWARD (1 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 ctstate INVALID ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ mark match 0x4000/0x4000 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED Chain KUBE-KUBELET-CANARY (0 references) target prot opt source destination Chain KUBE-PROXY-CANARY (0 references) target prot opt source destination Chain KUBE-SERVICES (3 references) target prot opt source destination 

    环境信息如下:

    [root@k8s-master ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.14", GitCommit:"0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", GitTreeState:"clean", BuildDate:"2021-08-11T18:07:41Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.14", GitCommit:"0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", GitTreeState:"clean", BuildDate:"2021-08-11T18:02:17Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"} [root@k8s-master ~]# cat /etc/redhat-release CentOS Linux release 7.9.2009 (Core) [root@k8s-master ]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready master 10d v1.19.14 k8s-node1 Ready <none> 10d v1.19.14 k8s-node2 Ready <none> 10d v1.19.14 k8s-node3 Ready <none> 10d v1.19.14 
    6 条回复    2021-09-03 08:57:29 +08:00
    defunct9
        1
    defunct9  
       2021-08-31 11:34:15 +08:00
    开 ssh,让我上去看看
    vemier
        2
    vemier  
       2021-08-31 12:31:07 +08:00
    @defunct9 老哥又来了
    nino789pzw
        3
    nino789pzw  
       2021-08-31 13:39:46 +08:00
    10.96.0.1 是什么的 ip ?
    你的 curl 是 insecure 的 command,ssh 到 Promethus 的 pod 上 ssh 到 10.96.0.1 试试?
    ice2016
        4
    ice2016  
    OP
       2021-08-31 21:10:10 +08:00
    @nino789pzw 把网络插件由 flannel 换成 calico 以后,正常部署了,也不报错,但是打开 svc
    对外暴露端口以后,一会正常一会就打不开了,过会又正常,,
    感觉哪里不对。。
    SIGEV13
        5
    SIGEV13  
       2021-08-31 22:53:19 +08:00
    老哥去开个 issue, 看看社区有没有懂得。
    ethangogogo
        6
    ethangogogo  
       2021-09-03 08:57:29 +08:00
    暴露端口之后有报错吗?
    关于     帮助文档     自助推广系统     博客     API     FAQ     Solana     5424 人在线   最高记录 6679       Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 37ms UTC 09:11 PVG 17:11 LAX 02:11 JFK 05:11
    Do have faith in what you're doing.
    ubao snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86