Spark提交到k8s解决问题记录

1.报错debian InRelease is not valid yet (invalid for another 2d 9h 30min 9s)

检查机器时间是不是没有同步，将时间配置为最新的时间。

2.PKIX path building failed

jdk证书问题。需要：

/* 
 * Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved. 
 * 
 * Redistribution and use in source and binary forms, with or without 
 * modification, are permitted provided that the following conditions 
 * are met: 
 * 
 *   - Redistributions of source code must retain the above copyright 
 *     notice, this list of conditions and the following disclaimer. 
 * 
 *   - Redistributions in binary form must reproduce the above copyright 
 *     notice, this list of conditions and the following disclaimer in the 
 *     documentation and/or other materials provided with the distribution. 
 * 
 *   - Neither the name of Sun Microsystems nor the names of its 
 *     contributors may be used to endorse or promote products derived 
 *     from this software without specific prior written permission. 
 * 
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS 
 * IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, 
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR 
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
 */  
  
import java.io.BufferedReader;  
import java.io.File;  
import java.io.FileInputStream;  
import java.io.FileOutputStream;  
import java.io.InputStream;  
import java.io.InputStreamReader;  
import java.io.OutputStream;  
import java.security.KeyStore;  
import java.security.MessageDigest;  
import java.security.cert.CertificateException;  
import java.security.cert.X509Certificate;  
  
import javax.net.ssl.SSLContext;  
import javax.net.ssl.SSLException;  
import javax.net.ssl.SSLSocket;  
import javax.net.ssl.SSLSocketFactory;  
import javax.net.ssl.TrustManager;  
import javax.net.ssl.TrustManagerFactory;  
import javax.net.ssl.X509TrustManager;  
  
public class InstallCert {  
  
    public static void main(String[] args) throws Exception {  
        String host;  
        int port;  
        char[] passphrase;  
        if ((args.length == 1) || (args.length == 2)) {  
            String[] c = args[0].split(":");  
            host = c[0];  
            port = (c.length == 1) ? 443 : Integer.parseInt(c[1]);  
            String p = (args.length == 1) ? "changeit" : args[1];  
            passphrase = p.toCharArray();  
        } else {  
            System.out  
                    .println("Usage: java InstallCert <host>[:port] [passphrase]");  
            return;  
        }  
  
        File file = new File("jssecacerts");  
        if (file.isFile() == false) {  
            char SEP = File.separatorChar;  
            File dir = new File(System.getProperty("java.home") + SEP + "lib"  
                    + SEP + "security");  
            file = new File(dir, "jssecacerts");  
            if (file.isFile() == false) {  
                file = new File(dir, "cacerts");  
            }  
        }  
        System.out.println("Loading KeyStore " + file + "...");  
        InputStream in = new FileInputStream(file);  
        KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());  
        ks.load(in, passphrase);  
        in.close();  
  
        SSLContext context = SSLContext.getInstance("TLS");  
        TrustManagerFactory tmf = TrustManagerFactory  
                .getInstance(TrustManagerFactory.getDefaultAlgorithm());  
        tmf.init(ks);  
        X509TrustManager defaultTrustManager = (X509TrustManager) tmf  
                .getTrustManagers()[0];  
        SavingTrustManager tm = new SavingTrustManager(defaultTrustManager);  
        context.init(null, new TrustManager[] { tm }, null);  
        SSLSocketFactory factory = context.getSocketFactory();  
  
        System.out  
                .println("Opening connection to " + host + ":" + port + "...");  
        SSLSocket socket = (SSLSocket) factory.createSocket(host, port);  
        socket.setSoTimeout(10000);  
        try {  
            System.out.println("Starting SSL handshake...");  
            socket.startHandshake();  
            socket.close();  
            System.out.println();  
            System.out.println("No errors, certificate is already trusted");  
        } catch (SSLException e) {  
            System.out.println();  
            e.printStackTrace(System.out);  
        }  
  
        X509Certificate[] chain = tm.chain;  
        if (chain == null) {  
            System.out.println("Could not obtain server certificate chain");  
            return;  
        }  
  
        BufferedReader reader = new BufferedReader(new InputStreamReader(  
                System.in));  
  
        System.out.println();  
        System.out.println("Server sent " + chain.length + " certificate(s):");  
        System.out.println();  
        MessageDigest sha1 = MessageDigest.getInstance("SHA1");  
        MessageDigest md5 = MessageDigest.getInstance("MD5");  
        for (int i = 0; i < chain.length; i++) {  
            X509Certificate cert = chain[i];  
            System.out.println(" " + (i + 1) + " Subject "  
                    + cert.getSubjectDN());  
            System.out.println("   Issuer  " + cert.getIssuerDN());  
            sha1.update(cert.getEncoded());  
            System.out.println("   sha1    " + toHexString(sha1.digest()));  
            md5.update(cert.getEncoded());  
            System.out.println("   md5     " + toHexString(md5.digest()));  
            System.out.println();  
        }  
  
        System.out  
                .println("Enter certificate to add to trusted keystore or 'q' to quit: [1]");  
        String line = reader.readLine().trim();  
        int k;  
        try {  
            k = (line.length() == 0) ? 0 : Integer.parseInt(line) - 1;  
        } catch (NumberFormatException e) {  
            System.out.println("KeyStore not changed");  
            return;  
        }  
  
        X509Certificate cert = chain[k];  
        String alias = host + "-" + (k + 1);  
        ks.setCertificateEntry(alias, cert);  
  
        OutputStream out = new FileOutputStream("jssecacerts");  
        ks.store(out, passphrase);  
        out.close();  
  
        System.out.println();  
        System.out.println(cert);  
        System.out.println();  
        System.out  
                .println("Added certificate to keystore 'jssecacerts' using alias '"  
                        + alias + "'");  
    }  
  
    private static final char[] HEXDIGITS = "0123456789abcdef".toCharArray();  
  
    private static String toHexString(byte[] bytes) {  
        StringBuilder sb = new StringBuilder(bytes.length * 3);  
        for (int b : bytes) {  
            b &= 0xff;  
            sb.append(HEXDIGITS[b >> 4]);  
            sb.append(HEXDIGITS[b & 15]);  
            sb.append(' ');  
        }  
        return sb.toString();  
    }  
  
    private static class SavingTrustManager implements X509TrustManager {  
  
        private final X509TrustManager tm;  
        private X509Certificate[] chain;  
  
        SavingTrustManager(X509TrustManager tm) {  
            this.tm = tm;  
        }  
  
        public X509Certificate[] getAcceptedIssuers() {  
            throw new UnsupportedOperationException();  
        }  
  
        public void checkClientTrusted(X509Certificate[] chain, String authType)  
                throws CertificateException {  
            throw new UnsupportedOperationException();  
        }  
  
        public void checkServerTrusted(X509Certificate[] chain, String authType)  
                throws CertificateException {  
            this.chain = chain;  
            tm.checkServerTrusted(chain, authType);  
        }  
    }  
  
}  

java InstallCert localhost:6443

输入1回车后成功生成证书，将生成在同目录下的jssecacerts的文件，复制到$JAVA_HOME/jre/lib/security目录下。

3./root/.kube/config file not found

如果使用的是k3s（k8s的裁剪版），那么运行命令：

cp /etc/rancher/k3s/k3s.yaml /root/.kube/config

4.Caused by: io.fabric8.kubernetes.client.KubernetesClientException: JcaPEMKeyConverter is provided by BouncyCastle, an optional dependency. To use support for EC Keys you must explicitly add this dependency to classpath.

引入如下的三个包：

 并在java.security文件中加入红框配置：

 

单机部署：

k3s的master默认也可以作为node，可以执行pod:
export K3S_NODE_NAME=localhost
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_EXEC="server --docker" sh -

token所在文件路径：/var/lib/rancher/k3s/server/node-token
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

新建spark用户：
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default

注意，有时候会报java.net.UnknownHostException: kubernetes.default.svc
使用iptables -F将防火墙禁用即可

提交spark任务：
./spark-submit  \
    --master k8s://https://localhost:6443  \
    --deploy-mode cluster  \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark  \
    --conf spark.kubernetes.container.image=spark:spark-docker  \
    --class org.apache.spark.examples.SparkPi  \
    --name sparkpi  \
    local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar

 

提交执行：

 

提交完毕后，可以在logs里看到pi计算的值：