golang

使用golang开发，导包的时候，需要配置go包代理，这里用七牛云的，稳定 七牛云 - Goproxy.cn

windows开发vscode开发环境

1，配置go mod的环境变量，以及go代理

系统环境变量里

GO111MODULE配置成on

GOPROXY配置成https://goproxy.cn

2，有了go代理就可以安装vscode的提示下载各种工具和包了

3，创建go module ，使用命令go mod init  module_name

然后使用go mod tidy  可以下载相应的包

4,在 go env的输出里

GOMODCACHE=C:\Users\admin\go\pkg\mod 

GOMODCACHE字段定义了所有包的源码位置，go mod tidy 就是把代码下载到这个位置

5，还有一个go.sum文件 用hash code校验代码包的正确性，防止代码被改动过

6，vendor的概念

go mod vendor 命令与go mod tidy相对应。go mod tidy是下载到GOMODCACHE指定的工作空间目录。而go mod vendor是直接把代码下载到项目源码目录里

api使用速查：

0，errors包，有一个errorString结构(或者说对象)，有一个New函数(也可以说是面向对象的set方法)，有一个Error函数(也可以说是面向对象的get方法)

基本上，os包，io包，filepath包，http，都是一起联动使用的，从操作系统，文件系统，到输入输出，到文件路径，到网络。

一般是先os open os打开，然后到io的writer和reader，io的writer和reader给到网络或者其他文件系统，网络的另一端服务端通过io.copy()可以传递到不同的io及文件系统

1，filepath包，

顾名思义，专门处理文件路径相关的，比如获取相对路径，绝对路径，获取文件名，文件扩展名，获取目录名，合成组装文件名

使用自定义函数递归遍历目录：

filepath.WalkDir   

WalkDir 是不遍历软连接的。

filepath.WalkDir来实现文件和目录的浏览。
filepath.WalkDir需要输入待浏览的目录，以及一个回调函数：

回调函数原型：

type WalkDirFunc func(path string, d DirEntry, err error) error

回调函数的返回值控制着遍历流程。如果回调函数返回SkipDir,跳过当前目录，其他情况如果回调函数返回一个error，停止遍历并将此error上报。

回调函数的error参数，开发者可以自己控制，到底是略过这个错误，还是上报这个错误，如果上报这个错误，遍历就停止了，如果略过这个错误比如即使err != nil也返回nil，遍历继续(也相当于skipdir了？)。剩下用户就可以应用自己的业务逻辑了，比如下面例子里的计数，递归统计整个目录下有多少文件。

两种情况下，error参数是非nil的，os.Lstat调用出错，Readdirnames method方法出错。

// The error result returned by the function controls how WalkDir

// continues. If the function returns the special value SkipDir, WalkDir

// skips the current directory (path if d.IsDir() is true, otherwise

// path's parent directory). Otherwise, if the function returns a non-nil

// error, WalkDir stops entirely and returns that error.

The err argument reports an error related to path, signaling that Walk

// will not walk into that directory. The function can decide how to

// handle that error; as described earlier, returning the error will

// cause Walk to stop walking the entire tree.

Walk calls the function with a non-nil err argument in two cases.

//

// First, if an os.Lstat on the root directory or any directory or file

// in the tree fails, Walk calls the function with path set to that

// directory or file's path, info set to nil, and err set to the error

// from os.Lstat.

//

// Second, if a directory's Readdirnames method fails, Walk calls the

// function with path set to the directory's path, info, set to an

// fs.FileInfo describing the directory, and err set to the error from

// Readdirnames.

type WalkDirFunc func(path string, d DirEntry, err error) error 

fn将在传入的目录中的每个文件和子目录上被调用。下面是一个计算当前目录中所有文件数量的示例。

import (
    "fmt"
    "io/fs"
    "path/filepath" ) // example of counting all files in a root directory func countFiles() int { count := 0; filepath.WalkDir(".", func(path string, file fs.DirEntry, err error) error { if err != nil { return err } if !file.IsDir() { fmt.Println(path); count++; } return nil; }); return count; }

WalkDir  源码注释：

// WalkDirFunc is the type of the function called by WalkDir to visit

// each file or directory.

//

// The path argument contains the argument to WalkDir as a prefix.

// That is, if WalkDir is called with root argument "dir" and finds a file

// named "a" in that directory, the walk function will be called with

// argument "dir/a".

//

// The d argument is the fs.DirEntry for the named path.

//

// The error result returned by the function controls how WalkDir

// continues. If the function returns the special value SkipDir, WalkDir

// skips the current directory (path if d.IsDir() is true, otherwise

// path's parent directory). Otherwise, if the function returns a non-nil

// error, WalkDir stops entirely and returns that error.

//

// The err argument reports an error related to path, signaling that

// WalkDir will not walk into that directory. The function can decide how

// to handle that error; as described earlier, returning the error will

// cause WalkDir to stop walking the entire tree.

//

// WalkDir calls the function with a non-nil err argument in two cases.

//

// First, if the initial fs.Stat on the root directory fails, WalkDir

// calls the function with path set to root, d set to nil, and err set to

// the error from fs.Stat.

//

// Second, if a directory's ReadDir method fails, WalkDir calls the

// function with path set to the directory's path, d set to an

// fs.DirEntry describing the directory, and err set to the error from

// ReadDir. In this second case, the function is called twice with the

// path of the directory: the first call is before the directory read is

// attempted and has err set to nil, giving the function a chance to

// return SkipDir and avoid the ReadDir entirely. The second call is

// after a failed ReadDir and reports the error from ReadDir.

// (If ReadDir succeeds, there is no second call.)

//

// The differences between WalkDirFunc compared to filepath.WalkFunc are:

//

//   - The second argument has type fs.DirEntry instead of fs.FileInfo.

//   - The function is called before reading a directory, to allow SkipDir

//     to bypass the directory read entirely.

//   - If a directory read fails, the function is called a second time

//     for that directory to report the error.

//

type WalkDirFunc func(path string, d DirEntry, err error) error

// walkDir recursively descends path, calling walkDirFn.

func walkDir(fsys FS, name string, d DirEntry, walkDirFn WalkDirFunc) error {

    if err := walkDirFn(name, d, nil); err != nil || !d.IsDir() {

        if err == SkipDir && d.IsDir() {

            // Successfully skipped directory.

            err = nil

        }

        return err

    }

    dirs, err := ReadDir(fsys, name)

    if err != nil {

        // Second call, to report ReadDir error.

        err = walkDirFn(name, d, err)

        if err != nil {

            return err

        }

    }

    for _, d1 := range dirs {

        name1 := path.Join(name, d1.Name())

        if err := walkDir(fsys, name1, d1, walkDirFn); err != nil {

            if err == SkipDir {

                break

            }

            return err

        }

    }

    return nil

}

// WalkDir walks the file tree rooted at root, calling fn for each file or

// directory in the tree, including root.

//

// All errors that arise visiting files and directories are filtered by fn:

// see the fs.WalkDirFunc documentation for details.

//

// The files are walked in lexical order, which makes the output deterministic

// but requires WalkDir to read an entire directory into memory before proceeding

// to walk that directory.

//

// WalkDir does not follow symbolic links found in directories,

// but if root itself is a symbolic link, its target will be walked.

func WalkDir(fsys FS, root string, fn WalkDirFunc) error {

    info, err := Stat(fsys, root)

    if err != nil {

        err = fn(root, nil, err)

    } else {

        err = walkDir(fsys, root, &statDirEntry{info}, fn)

    }

    if err == SkipDir {

        return nil

    }

    return err

}

源码以及注释说明的非常清楚：回调函数是控制整个遍历行为的：

The caller's behavior is controlled by the return value, which is decided

        // by walkFn. walkFn may ignore err and return nil.

        // If walkFn returns SkipDir, it will be handled by the caller.

        // So walk should return whatever walkFn returns.

// walk recursively descends path, calling walkFn.

func walk(path string, info fs.FileInfo, walkFn WalkFunc) error {

    if !info.IsDir() {

        return walkFn(path, info, nil)

    }

    names, err := readDirNames(path)

    err1 := walkFn(path, info, err)

    // If err != nil, walk can't walk into this directory.

    // err1 != nil means walkFn want walk to skip this directory or stop walking.

    // Therefore, if one of err and err1 isn't nil, walk will return.

    if err != nil || err1 != nil {

        // The caller's behavior is controlled by the return value, which is decided

        // by walkFn. walkFn may ignore err and return nil.

        // If walkFn returns SkipDir, it will be handled by the caller.

        // So walk should return whatever walkFn returns.

        return err1

    }

    for _, name := range names {

        filename := Join(path, name)

        fileInfo, err := lstat(filename)

        if err != nil {

            if err := walkFn(filename, fileInfo, err); err != nil && err != SkipDir {

                return err

            }

        } else {

            err = walk(filename, fileInfo, walkFn)

            if err != nil {

                if !fileInfo.IsDir() || err != SkipDir {

                    return err

                }

            }

        }

    }

    return nil

}

os包：

os.open()   返回文件描述符

2，io包： 顾名思义，专门处理输入输出相关的。

2.1

  io.copy(目标，源) ，流式在不同io之间传递数据

2.2

io.Pipe() (*PipeReader, *PipeWriter)

reader writer交替运行。也就是说，Pipe适用于，产生了一条数据，紧接着就要处理掉这条数据的场景。而且因为其内部是一把大锁，因此是线程安全的。

2.3

MultiWriter()   方法，类型Linux的tee命令，一个io输入同时写到多个io输出

3，flag包，处理命令行相关的

命令行使用：

go run main.go -dir=/opt -port=3306

1，直接取相应类型的值类型到变量。先定义变量，然后获取使用

var (

    parallel       uint

    port            uint

    dir            string

)

func main() {

    flag.StringVar(&dir, "dir", "", "目录的路径")  字符串类型

    flag.UintVar(&port,"port",8080."监听的端口号")  监听的端口号 命令行使用-port=3306,默认是8080 端口号用无符号整型

    flag.UintVar(&parallel, "parallel", 1, "并发数默认为1个并发") -parallel=1

    flag.Parse()

}

4，net/http  处理http网络相关的包: http package - net/http - pkg.go.dev

The client must close the response body when finished with it:

resp, err := http.Get("http://example.com/")
if err != nil {
	// handle error
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)

For control over HTTP client headers, redirect policy, and other settings, create a Client:

client := &http.Client{
	CheckRedirect: redirectPolicyFunc,
}

resp, err := client.Get("http://example.com")
// ...

req, err := http.NewRequest("GET", "http://example.com", nil)
// ...
req.Header.Add("If-None-Match", `W/"wyzzy"`)
resp, err := client.Do(req)
// ...

For control over proxies, TLS configuration, keep-alives, compression, and other settings, create a Transport:

tr := &http.Transport{
	MaxIdleConns:       10,
	IdleConnTimeout:    30 * time.Second,
	DisableCompression: true,
}
client := &http.Client{Transport: tr}
resp, err := client.Get("https://example.com")

ListenAndServe starts an HTTP server with a given address and handler. The handler is usually nil, which means to use DefaultServeMux. Handle and HandleFunc add handlers to DefaultServeMux:

http.Handle("/foo", fooHandler)

http.HandleFunc("/bar", func(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
})

log.Fatal(http.ListenAndServe(":8080", nil))

More control over the server's behavior is available by creating a custom Server:

s := &http.Server{
	Addr:           ":8080",
	Handler:        myHandler,
	ReadTimeout:    10 * time.Second,
	WriteTimeout:   10 * time.Second,
	MaxHeaderBytes: 1 << 20,
}
log.Fatal(s.ListenAndServe())

所谓文件上传，就是用一个文件io，向http server发送请求，

5,io/ioutil(1.16版本已经废弃这个包了，转移到io和os这两个包了)

io工具包

ioutil包的方法

// Discard 是一个io.Writer 接口，调用它的Write方法将不做任何事情
// 并且始终成功返回
var Discard io.Writer = devNull(0)

// ReadAll读取r中的所有数据，返回读取的数据和遇到的错误 // 如果读取成功，则err返回nil，而不是EOF，因为ReadAll定义为读取 // 所有数据，所以不会把EOF当作错误处理 func ReadAll(r io.Reader) ([]byte, error) // ReadFile读取文件中的所有数据，返回读取的数据和遇到的错误 // 如果读取成功，则err返回nil，而不是EOF func ReadFile(filename string) ([]byte, error) // WriteFile向文件中写入数据，写入前会清空文件。 // 如果文件不存在，则会以指定的权限创建文件 // 返回遇到的错误 func WriteFile(filename string, data []byte, perm os.FileMode) error // ReadDir 读取指定目录中的所有目录和文件(不包括子目录) // 返回读取到文件信息列表和遇到错误，列表是经过排序的。 func ReadDir(dirname string) ([]os.FileInfo, error) // NopCloser 将r包装成一个ReadCloser类型，但Close方法不做任何事情 func NopCloser(r io.Reader) io.ReadCloser // TempFile在dir目录中创建一个以prefix为前缀的临时文件，并将其以读写模式打开。返回创建的文件对象和遇到的错误 // 如果dir为空，则在默认的临时目录中创建文件(参加os.TempDir)，多次调用会创建不通的临时文件，调用者可以通过f.Name()获取文件的完整路径 // 调用本函数所创建的临时文件，应该由调用者字节删除 func TempFile(dir, prefix string) (f *os.File, err error) // TempDir功能通TempFile，只不过创建的是目录，返回目录的完整路径 func TempDir(dir, prefix string) (name string, err error)


yaml package - gopkg.in/yaml.v3 - pkg.go.dev
序列化:把对象或者说结构体转化为字节序列，可以用yaml.Marshal 
反序列化：把字符序列转化为对象或者说结构体，可以yaml.Unmarshal

7,os/signal 包。处理操作系统信号包
信号和信号量，是操作系统里的概念，用处是用来同步进程的状态。响应的标准和api是：

go中的信号量
信号 值 动作 说明
SIGHUP 1 Term 终端控制进程结束(终端连接断开)
SIGINT 2 Term 用户发送INTR字符(Ctrl+C)触发
SIGQUIT 3 Core 用户发送QUIT字符(Ctrl+/)触发
SIGILL 4 Core 非法指令(程序错误、试图执行数据段、栈溢出等)
SIGABRT 6 Core 调用abort函数触发
SIGFPE 8 Core 算术运行错误(浮点运算错误、除数为零等)
SIGKILL 9 Term 无条件结束程序(不能被捕获、阻塞或忽略)
SIGSEGV 11 Core 无效内存引用(试图访问不属于自己的内存空间、对只读内存空间进行写操作)
SIGPIPE 13 Term 消息管道损坏(FIFO/Socket通信时，管道未打开而进行写操作)
SIGALRM 14 Term 时钟定时信号
SIGTERM 15 Term 结束程序(可以被捕获、阻塞或忽略)
SIGUSR1 30,10,16 Term 用户保留
SIGUSR2 31,12,17 Term 用户保留
SIGCHLD 20,17,18 Ign 子进程结束(由父进程接收)
SIGCONT 19,18,25 Cont 继续执行已经停止的进程(不能被阻塞)
SIGSTOP 17,19,23 Stop 停止进程(不能被捕获、阻塞或忽略) SIGTSTP 18,20,24 Stop 停止进程(可以被捕获、阻塞或忽略) SIGTTIN 21,21,26 Stop 后台程序从终端中读取数据时触发 SIGTTOU 22,22,27 Stop 后台程序向终端中写数据时触发
有些信号名对应着3个信号值，这是因为这些信号值与平台相关
SIGKILL和SIGSTOP这两个信号既不能被应用程序捕获，也不能被操作系统阻塞或忽略

在go语言中：

var (

    Interrupt Signal = syscall.SIGINT

    Kill      Signal = syscall.SIGKILL

)

    go app.Run()  比如用go程启动一个server

    stop := make(chan os.Signal)   创建一个接收信号的chan

    signal.Notify(stop, os.Kill, os.Interrupt) 注册监听捕获接收Kill信号和Interrupt信号(如果不指定信号，将接收全部信号)，就是操作系统的SIGINT和SIGKILL信号，

也就是2信号和9信号。那2信号和9信号分别代表什么呢，和15信号一起来说：都是退出分别不同是：2相当与CTRL+C，9直接杀死，15优雅退出在杀死之前允许clean up做一些清理动作。

    <-stop 阻塞一直等待stop chan里的数据，也就是2信号和9信号，一有数据，说明开发者对操作系统发出了这两个信号，想要结束程序，那么就可以执行app.stop()了

    app.Stop()

    logger.Infof("[main.go::main] shutdown")

信号编号大全：

附录：

linux signals

Signal Name	Number	Description
SIGHUP	1	Hangup (POSIX)
SIGINT	2	Terminal interrupt (ANSI)
SIGQUIT	3	Terminal quit (POSIX)
SIGILL	4	Illegal instruction (ANSI)
SIGTRAP	5	Trace trap (POSIX)
SIGIOT	6	IOT Trap (4.2 BSD)
SIGBUS	7	BUS error (4.2 BSD)
SIGFPE	8	Floating point exception (ANSI)
SIGKILL	9	Kill(can't be caught or ignored) (POSIX)
SIGUSR1	10	User defined signal 1 (POSIX)
SIGSEGV	11	Invalid memory segment access (ANSI)
SIGUSR2	12	User defined signal 2 (POSIX)
SIGPIPE	13	Write on a pipe with no reader, Broken pipe (POSIX)
SIGALRM	14	Alarm clock (POSIX)
SIGTERM	15	Termination (ANSI)
SIGSTKFLT	16	Stack fault
SIGCHLD	17	Child process has stopped or exited, changed (POSIX)
SIGCONT	18	Continue executing, if stopped (POSIX)
SIGSTOP	19	Stop executing(can't be caught or ignored) (POSIX)
SIGTSTP	20	Terminal stop signal (POSIX)
SIGTTIN	21	Background process trying to read, from TTY (POSIX)
SIGTTOU	22	Background process trying to write, to TTY (POSIX)
SIGURG	23	Urgent condition on socket (4.2 BSD)
SIGXCPU	24	CPU limit exceeded (4.2 BSD)
SIGXFSZ	25	File size limit exceeded (4.2 BSD)
SIGVTALRM	26	Virtual alarm clock (4.2 BSD)
SIGPROF	27	Profiling alarm clock (4.2 BSD)
SIGWINCH	28	Window size change (4.3 BSD, Sun)
SIGIO	29	I/O now possible (4.2 BSD)
SIGPWR	30	Power failure restart (System V)

关于2 9 15  这里写的不错：
https://www.baeldung.com/linux/sigint-and-other-termination-signals

How SIGINT Relates to SIGTERM, SIGQUIT and SIGKILL

Now that we understand more about signals, we can see how they relate to each other.

The default action for SIGINT, SIGTERM, SIGQUIT, and SIGKILL is to terminate the process. However, SIGTERM, SIGQUIT, and SIGKILL are defined as signals to terminate the process, but SIGINT is defined as an interruption requested by the user.

In particular, if we send SIGINT (or press Ctrl+C) depending on the process and the situation it can behave differently. So, we shouldn’t depend solely on SIGINT to finish a process.

As SIGINT is intended as a signal sent by the user, usually the processes communicate with each other using other signals. For instance, a parent process usually sends SIGTERM to its children to terminate them, even if SIGINT has the same effect.

In the case of SIGQUIT, it generates a core dump which is useful for debugging.

Now that we have this in mind, we can see we should choose SIGTERM on top of SIGKILL to terminate a process. SIGTERM is the preferred way as the process has the chance to terminate gracefully.

As a process can override the default action for SIGINT, SIGTERM, and SIGQUIT, it can be the case that neither of them finishes the process. Also, if the process is hung it may not respond to any of those signals. In that case, we have SIGKILL as the last resort to terminate the process.

 -----------------------------------------------------------------------------------------------------------------------------------------------

package main

import (

    "flag"

    "fmt"

    "log-forward/conf"

    "log-forward/kafka"

    "log-forward/us3"

    "time"

    "gopkg.in/ini.v1"

)

var config = new(conf.Config)

var kafkaMsg = make(chan string)

func main() {

    var (

        confFile    string

        us3ConfFile string

    )

    flag.StringVar(&confFile, "confFile", "config.ini", "set config file")

    flag.StringVar(&us3ConfFile, "us3ConfFile", "config.json", "set the us3 config json file")

    flag.Parse()

    err := ini.MapTo(config, confFile)

    if err != nil {

        fmt.Printf("Fail to read file: %v", err)

        return

    }

    fmt.Println(config)

    if _, err = kafka.Init(config.Kafka.Dsn); err != nil {

        fmt.Println("init kafka failed, err:%v\n", err)

        return

    }

    fmt.Println("init kafka success.")

    us3.Run(us3ConfFile, kafkaMsg)

    run()

}

func run() {

    for {

        select {

        case line := <-kafkaMsg:

            kafka.SendToKafka(config.Kafka.Topic, line)

        default:

            time.Sleep(time.Second)

        }

    }

}

conf/config.go

package conf

type Config struct {

    Kafka   Kafka   `ini:"kafka"`

}

type Kafka struct {

    Dsn   string `ini:"dsn"`

    Topic string `ini:"topic"`

}

conf/config.ini

[kafka]

dsn=kafka://username:pasword@ip1:9093,ip2:9093,ip3:9093

topic=vsulblog

kafka/kafka.go

package kafka

import (

    "crypto/sha256"

    "fmt"

    "strings"

    "github.com/Shopify/sarama"

    "github.com/wjiec/gdsn"

    "github.com/xdg-go/scram"

)

var (

    client sarama.SyncProducer

)

var (

    SHA256 scram.HashGeneratorFcn = sha256.New

)

type XDGSCRAMClient struct {

    *scram.Client

    *scram.ClientConversation

    scram.HashGeneratorFcn

}

func (x *XDGSCRAMClient) Begin(userName, password, authzID string) (err error) {

    x.Client, err = x.HashGeneratorFcn.NewClient(userName, password, authzID)

    if err != nil {

        return err

    }

    x.ClientConversation = x.Client.NewConversation()

    return nil

}

func (x *XDGSCRAMClient) Step(challenge string) (response string, err error) {

    response, err = x.ClientConversation.Step(challenge)

    return

}

func (x *XDGSCRAMClient) Done() bool {

    return x.ClientConversation.Done()

}

func Init(kafkaDSN string) (sarama.SyncProducer, error) {

    d, err := gdsn.Parse(kafkaDSN)

    if err != nil {

        return nil, err

    }

    config := sarama.NewConfig()

    config.Producer.RequiredAcks = sarama.WaitForAll

    config.Producer.Return.Successes = true

    if d.User.Username() != "" {

        config.Metadata.Full = true

        config.Net.SASL.Enable = true

        config.Net.SASL.User = d.User.Username()

        config.Net.SASL.Password, _ = d.User.Password()

        config.Net.SASL.Handshake = true

        config.Net.SASL.SCRAMClientGeneratorFunc = func() sarama.SCRAMClient {

            return &XDGSCRAMClient{

                HashGeneratorFcn: SHA256,

            }

        }

        config.Net.SASL.Mechanism = sarama.SASLTypeSCRAMSHA256

    }

    client, err = sarama.NewSyncProducer(strings.Split(d.Address(), ","), config)

    if err != nil {

        fmt.Println("producer closed, err:", err)

    }

    return client, err

}

func SendToKafka(topic, data string) {

    msg := &sarama.ProducerMessage{}

    msg.Topic = topic

    msg.Value = sarama.StringEncoder(data)

    pid, offset, err := client.SendMessage(msg)

    if err != nil {

        fmt.Println("sned mage failed, err:", err)

    }

    fmt.Printf("pid:%v offset:%v\n", pid, offset)

    fmt.Println("send ok")

}

us3/us3.go

package us3

import (

    "bytes"

    "compress/gzip"

    "io"

    "log"

    "strings"

    "time"

    ufsdk "github.com/ufilesdk-dev/ufile-gosdk"

)

const (

    delim byte = '\n'

)

func downloadFile(filename string, req *ufsdk.UFileRequest) (buf bytes.Buffer) {

    // 下载文件

    log.Println("下载文件: ")

    reqUrl := req.GetPrivateURL(filename, 10*time.Second)

    err := req.Download(reqUrl)

    if err != nil {

        log.Fatalln(string(req.DumpResponse(true)))

    }

    log.Printf("下载文件成功！")

    log.Println("print file context:")

    //log.Println(string(req.LastResponseBody))

    us3ResponseBodyReader := strings.NewReader(string(req.LastResponseBody))

    gzipReader, err := gzip.NewReader(us3ResponseBodyReader)

    if err != nil {

        log.Println("get gzipreader fail")

        return

    }

    defer gzipReader.Close()

    _, err = io.Copy(&buf, gzipReader)

    if err != nil {

        log.Println("io copy gzipreader fail")

        return

    }

    log.Println(buf.String())

    return buf

}

func sendFileLog(buf bytes.Buffer, kafkaMsg chan string) {

    go func() {

        for {

            if len(buf.String()) == 0 {

                break

            }

            logLine, err := buf.ReadString(delim)

            if err != nil {

                log.Println("read log file err")

                log.Println(err)

            }

            kafkaMsg <- logLine

            continue

        }

    }()

}

func Run(ConfigFile string, kafkaMsg chan string) {

    // 准备下载请求与要下载的文件

    config, err := ufsdk.LoadConfig(ConfigFile)

    if err != nil {

        panic(err.Error())

    }

    req, err := ufsdk.NewFileRequest(config, nil)

    if err != nil {

        panic(err.Error())

    }

    marker := ""

    var buf bytes.Buffer

    for {

        //req.PrefixFileList must have marker param, and if it is "",it will get data from begin.

        //so if the NextMarker is "",we assign the current filename to the marker

        myPrefixFileList, err := req.PrefixFileList("ulb-xim0mgmm", marker, 1)

        if err != nil {

            log.Println("DumpResponse：", string(req.DumpResponse(true)))

        }

        if myPrefixFileList.NextMarker == "" {

            log.Println(marker)

            log.Println()

            buf = downloadFile(myPrefixFileList.DataSet[0].FileName, req)

            sendFileLog(buf, kafkaMsg)

            marker = myPrefixFileList.DataSet[0].FileName

            log.Println("last file marker change")

            time.Sleep(360 * time.Second) //ucloud dump ulb log to us3 Every five minutes

            continue

        } else {

            log.Println(marker)

            log.Println()

            buf = downloadFile(myPrefixFileList.DataSet[0].FileName, req)

            sendFileLog(buf, kafkaMsg)

            marker = myPrefixFileList.NextMarker

        }

    }

}

-------------------------------------------------

go包

解析ini配置，ini package - gopkg.in/ini.v1 - pkg.go.dev

类似Linux下的tail命令，tail package - github.com/hpcloud/tail - pkg.go.dev

kafka库，sarama package - github.com/Shopify/sarama - pkg.go.dev

dsn解析包，https://github.com/wjiec/gdsn

-----------------------------------------------------------------------------------------------------

go tips

判断字符串是否为空：

str := ""
fmt.Println(str == "")
fmt.Println(len(str ) == 0)

str == "" 和 len(str) == 0 都可以，而且编译成汇编之后代码是一样的，见https://www.jianshu.com/p/1180bdcb2e48，这个老哥专研精神不错。