Showing posts tagged 'golang'

Release of Scalingo CLI 1.2.0

Another load of new features for Scalingo command line utility. We’re introducing here the first brick of our application monitoring stack, the subcommand ‘stats’. Alongside this new feature, the UX has been improved for the ‘logs’, ‘run’ and ‘db-tunnel’ commands. These changes will help you being more efficient when managing your applications.

image

Scalingo CLI 1.1.0: smart completion and improved UX

Update : Scalingo CLI 1.1.1 has been released, look at the end of the post for more details.

During these past weeks, the command line client scalingo has been greatly improved, adding new features that will enhance your working environment and productivity. These features include the smart completion of every command and subcommand, a --remote flag which helps you selecting the environment of your choice, and a --filter flag available to the “logs” command.

image

#

Graceful server restart with Go

Go has been designed as a backend language and is mostly used as such. Servers are the most common type of software produced with it. The question I’m going to answer here is: how to cleanly upgrade a running server?

image

Goals:

  • Do not close any of the existing connections: for instance, we don’t want to cut down any running deployment. However we want to be able to upgrade our services whenever we want without any constraint.
  • The socket should always be available for the users: if the socket is unavailable at any moment some user may get a ‘connection refused’ message which is not acceptable.
  • The new version of the process should be started and should replace the old one.

Writing a replacement to OpenSSH using Go (2/2)

Last week, we have seen that tools exist to build SSH based applications with Go:

This week, we are going to give advanced usage examples of the golang.org/x/crypto/ssh package.

image

Appsdeck use case

To deploy applications on our platform, people are using GIT over SSH. The front-end SSH server has the following responsabilities:

  • Authenticate the user
  • Check that the user want to execute a git command
  • Ensure the user can access the requested repository
  • Forward the connection to a server which will manage the application deployment

For each of these items, I will give an example about how it can be done with Go.

Authentication

During the server setup, it is required to build a ssh.ServerConfig struct defining the different accepted ways to authenticate:

  • PublicKeyCallback authenticate with a private/public key pair
  • PasswordCallback: authenticate with a simple password
  • KeyboardInteractiveCallback: create challenges that the user has to answer interactively

The one we are using is PublicKey authentication, the callback handler is straightforward:

// Server setup:
config := ssh.ServerConfig{
PublicKeyCallback: keyAuthCallback,
}

// keyAuthCallback is called when a user tries to authenticate with a private/public key pair.
func keyAuthCallback(conn ssh.ConnMetadata, key ssh.PublicKey) (*ssh.Permissions, error) {
user, err := auth.AuthenticateUser(conn.User(), key)
if err != nil {
log.Println("Fail to authenticate", conn, ":", err)
return nil, errors.New("invalid authentication")
}

return &ssh.Permissions{Extensions: map[string]string{"user_id": user.Id}}, nil
}

All the cryptographic functions are handled by the package, the callback is used to identify users against any data source. It is possible to build anything behind auth.AuthenticateUser, a redis backend, a specific API. The first goal has been reached, we can use our own authentication system.

As you can see, I’m using the ssh.Permissions.Extensions map to keep track of the real user ID which has been sucessfully authenticated, this struct will be attached to the ssh.Conn object, so it’s a good way to pass metadata.

Limit user actions

The second goal is to check if the user is doing something he is allowed to. In our case, we reduce the scope to the execution of git-receive-pack and git-upload-pack.

To allow multiplexing into a single connection, SSH has a ‘channel’ system, multiple channels can exist concurrently. Each channel has a type and can request ‘actions’.

The type of channel which interests us is the ‘session’ channel, doing an ‘exec’ action. A complete example can be found on Github:

https://github.com/Scalingo/go-ssh-examples/blob/master/server_git.go

In the first function handleChanReq, we are filtering channel creation requests from the client and then read the first action request which should be ‘exec’.

func handleChanReq(chanReq ssh.NewChannel) {
if chanReq.ChannelType() != "session" {
chanReq.Reject(ssh.Prohibited, "channel type is not a session")
return
}

ch, reqs, err := chanReq.Accept()
if err != nil {
log.Println("fail to accept channel request", err)
return
}

req := <-reqs
if req.Type != "exec" {
ch.Write([]byte("request type '" + req.Type + "' is not 'exec'\r\n"))
ch.Close()
return
}

handleExec(ch, req)
}

When calling the second function, we get the action payload and check if the requested command is ’git-receive-pack’ (git push) or ’git-upload-pack’ (git fetch). If it is not, the channel is closed.

// handleExec filter the command which can be run.// Payload: string: command
func handleExec(ch ssh.Channel, req *ssh.Request) {
command := string(req.Payload)
gitCmds := []string{"git-receive-pack", "git-upload-pack"}

valid := false
for _, cmd := range gitCmds {
if strings.HasPrefix(command, cmd) {
valid = true
}
}
if !valid {
ch.Write([]byte("command is not a GIT command\r\n"))
ch.Close()
return
}

ch.Write([]byte("well done!\r\n"))
ch.Close()
}

Now we are sure that the user tries to run a git command.

Results:

└> ssh localhost -p 2222
request type 'pty-req' is not 'exec'
Connection to localhost closed.
└> ssh localhost -p 2222 ls
command is not a GIT command
└> ssh localhost -p 2222 git-receive-pack
well done!

Ensure the user can access the requested repository

Getting the executable name is good, but not enough. When someone runs git push appsdeck master with the following remote: ’git@appsdeck.eu:myapp.git’ the command is ’git-receive-pack myapp.git’, so there is a bit more work to do to analyse the command line.

But there is nothing here related SSH, we just have to parse the string and contact a backend to check if the user is allowed to deploy/fetch the given application.

Forward connection to the next host

The last step of our SSH server is to forward the connection to a host able to achieve the deployment. To do that, our server has to be invisible for the client. It does all the checking then, pipes the connection to the next host, as a simple reverse proxy.

So there are two steps:

  • Create a SSH connection to the target server
  • Forward the connection

SSH connection

In this part, we have to use golang.org/x/crypto/ssh package again, but this time from the client perspective:

func connectToHost(user, host string, key ssh.Signer) (*ssh.Client, *ssh.Session, error) {
sshConfig := &ssh.ClientConfig{
User: use,
Auth: []ssh.AuthMethod{
ssh.PublicKeys(key),
},
}

client, err := ssh.Dial("tcp", host, sshConfig)
if err != nil {
return err
}

session, err := client.NewSession()
if err != nil {
client.Close()
return err
}

return client, session, nil
}

The previous example connects to a server using the given SSH key and user, and creates a new ‘session’ channel, then, it returns this session which will be used forward and the associated connection, in order to be correctly closed later.

Connection forwarding

Go provides a really nice mecanism to do this pipe: io.Copy

targetStderr, _ := targetSession.StderrPipe()
targetStdout, _ := targetSession.StdoutPipe()
targetStdin, _ := targetSession.StdinPipe()

wg := &sync.WaitGroup{}
wg.Add(3)

go func() {
defer wg.Done()
io.Copy(targetStdin, userChannel)
}()

go func() {
defer wg.Done()
io.Copy(userChannel.Stderr(), targetStderr)
}()

go func() {
defer wg.Done()
io.Copy(userChannel, targetStdout)
}()

wg.Wait()

That’s it, the session has been completely piped from the user to the next server. When either the client or the upstream server closes the connection, the goroutines stop and the wg.Wait() instruction will be passed. This code does not have all the error checks, but do it on any program you intend to run in production, period! ;-)

Testing the proxy with OpenSSH

git clone https://github.com/Scalingo/go-ssh-examples
cd go-ssh-examples

# Generate host and user keypairs
bash init.sh

# Run an openssh server on the port 2223
/sbin/sshd -D -o Port=2223 -h `pwd`/host_key -o AuthorizedKeysFile=`pwd`/user_key.pub &

go run proxy/server.go proxy/connect_upstream.go &
ssh localhost -p 2222 <any command>

Conclusion

This article has been more technical than the previous one, We hope you enjoyed it. we wanted to share how Go could be used to work with SSH, in a fairly simple manner.

Don’t hesitate to ask questions if you need more details, we’ll be glad to answer them.

This article was the 3rd post of the serie #FridayTechnical, see you next week!

Writing a replacement to OpenSSH using Go (1/2)

The second part is live: Writing a replacement to OpenSSH using Go (2/2)

SSH is a well-known protocol to access remote servers and OpenSSH is the most common implementation. This article will explain how to build a SSH server and client using Go.

image

TL;DR

  • Why OpenSSH Server is not sufficient in a distributed environment
  • What are the available tools and libraries in Go to use the SSH protocol
  • A few implementation examples

Git and SSH

In a Platform as a Service such as Appsdeck, the standard way to deploy an application is to use git push. Underneath, git is simply using the ssh command. The two lines below explain the link between them:

# With appsdeck remote: git@appsdeck.eu:myproject.git
~/myproject $ git push appsdeck master

# Git will execute:
~/myproject $ ssh git@appsdeck.eu git-receive-pack myproject.git

Actually, it is possible to tell git to use another SSH client, by setting the environment variable GIT_SSH.

OpenSSH limitation

To authenticate a user to a remote server using OpenSSH Server, the user’s public SSH key has to be appended in the ~/.ssh/authorized_keys file of the target user. If OpenSSH finds the public key in this file, the authentication is considered successful. By default, it would execute the shell of the user, defined by its account.

It is possible to execute a custom command by prepending it to the public SSH key in the authorized_keys file:

command="ssh-handler" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAA[...]2bF26cl

In this case, instead of running the shell of the user, it will run the command “ssh-handler”. Our first prototype was using this feature to authenticate and authorize our users. However, as soon as you want to build a real distributed infrastructure, it’s not sufficient anymore. Indeed, with this method, it means that several hosts have to keep their authorized_keys file containing all the public keys of our users in sync. That’s two problems in one : we had to find a synchronization mechanism, and, sensitive data are spread out on several machines.

That’s why we decided it was time to build something more robust and scalable by writing our own custom SSH server, which would be able to use a custom backend to identify our users and then to forward the SSH connection to another host which will actually execute the GIT operation.

SSH in the Go world

We decided to write this custom SSH server with Go for different reasons. Except the fact that we had experience with the language, this official package (http://godoc.org/golang.org/x/crypto/ssh) implements everything we needed to build what we were looking for, while keeping it simple.

At Appsdeck, we are looking to respect the KISS principle. If a software is simple, it is easier to conserve readability, modularity and easy maintenance. By the way, that’s also why Go has become one of our core languages.

The SSH RFCs

While the Go package implements SSH connections setup (that includes transport and crypto), you still have to understand and to use SSH applicative protocol to build your software.

To achieve this, it is important to get how SSH is working, the following RFCs define the protocol:

Anytime you need to get explanations about a method or a constant of the Go package, refering to the RFCs is pretty straightforward with a simple text search.

(RFCs 4255 and 4256 also concern SSH but are less useful in this scope)

Code Examples

1. Basic server

The following code example defines a simple SSH server: https://github.com/Scalingo/go-ssh-examples/blob/master/server.go

This server only displays the kind of SSH key used for authentication (if any), accepts it, then closes the connection.

Server:

└> go run server.go
2014/12/05 15:41:27 [::1]:46428 authenticate with ssh-rsa
2014/12/05 15:41:27 Connection from [::1]:46428

Client:

└> ssh localhost -p 2222
Connection to localhost closed by remote host.
Connection to localhost closed.

As expected, our Go server succeeds to speak with OpenSSH, but then the connection is closed instantly.

2. Basic client

https://github.com/Scalingo/go-ssh-examples/blob/master/client.go

Usage:

go run client.go <user> <server:port> <command>

This client has only one job, it connects to a server, creates a sessions, executes a command, then prints the output and disconnects.

Output:

└> go run client.go foobar example.com:22 'ls /'
Password: *********
bin
boot
conf.d
dev
etc
home
initrd.img
lib[...]

Conclusion

This article introduces our problematic and how to use the SSH protocol with Go. In the next article we’ll explain how we use this library to solve our real problems (Authentication, Authorization, Connection Proxying).

To be continued…

… __Ready? Second part is here: Writing a replacement to OpenSSH using Go (2/2)

Links

This article was the 2nd post of the serie #FridayTechnical, see you next time!

Credits

Gopher image by Renee French (Creative Commons Attributions 3.0) Puffy - OpenSSH logo

— Léo Unbekandt, CTO @ Appsdeck

Async jobs between Go and Ruby

This post is the first article of the #FridayTechnical serie. We’ll try to post some interesting technical content about the languages, the libraries and the tools we use to build our product.

Our technological stack is mostly based on Ruby and Go. At some point, we had to make those languages communicate with each other. This article focuses on asynchronous jobs management between them.

image

In any infrastructure which is not composed of a single process, background workers are necessary. Sidekiq is a fairly popular tool to achieve this in the ruby ecosystem. It stores its state in a redis database, common to the clients and to the workers. At Appsdeck, our infrastructure is more complicated than that, and it is a requirement for us to send asynchronous tasks from service to service.

Go-Workers (https://github.com/jrallison/go-workers) is a Sidekiq-compatible tool to handle jobs with Go, using almost the same DSL. We’ll see later, that with some contributions we’ve done, it does its job quite well.

Worker

The worker job is to listen for tasks. It polls regularly the queues you have configured and executes the jobs it finds.

package main

import (
"fmt"

"github.com/Scalingo/go-workers"
)

func main() {
workers.Configure(map[string]string{
"process": "worker1", "password": "secret",
"server": "localhost:6379", "namespace": "goworkers",
}) workers.Process("myqueue", MyGoWorker, 10)
workers.Run()
}

func MyGoWorker(msg *workers.Msg) {
fmt.Println("running task", msg)
}

First, the redis connection and the process identity have to be configured. Then, one or several calls to the Process method define the queues and the worker functions associated to them.

The worker method, receives a *workers.Msg as argument. It is actually a wrapper of a *simplejson.Json containing the arguments. 

require "sidekiq"

Sidekiq::configure_server do |config|
config.redis = { url: "redis://:secret@localhost:6379", namespace: "goworkers" }
end

Sidekiq.options[:queues] = ["myqueue"]

class MyRubyWorker
include Sidekiq::Worker

def perform(str)
puts "Hello from Sidekiq: #{str}"
end
end

The process is pretty similar with Sidekiq, except that the Worker classes are automatically registered when including Sidekiq::Worker.

Client

The client is the process which adds jobs to the queues. The configuration setup is similar, but instead of registering workers, jobs are enqueued in queues.

package main

import "github.com/Scalingo/go-workers"

func main() {
workers.Configure(map[string]string{
"process": "client1", "password": "secret",
"server": "localhost:6379", "namespace": "goworkers", })
workers.Enqueue("myqueue", "MyRubyWorker", []string{"hello"})
}

Here is the main difference between the standard Sidekiq workflow. Instead of using MyRubyWorker.enqueue "arg1", it is required to use Sidekiq::Client.push. The reason is simple, the classes are not defined in the Ruby program, but in the Go worker. (we could define empty classes, but it’s, in my opinion, clumsy.)

require "sidekiq"

Sidekiq::configure_client do |config|
config.redis = { url: "redis://:secret@localhost:6379", namespace: "goworkers" }
end

Sidekiq::Client.push "queue" => "myqueue", "class" => "MyGoWorker", "args" => ["hello"]

Job scheduling

The package jrallison/go-workers is actually working but is lacking some features, especially, scheduled jobs enqueueing. For instance, if you want to send an email to your users at 9 in the morning, it is important to have this feature. Our fork (https://github.com/Scalingo/go-workers) of the project (waiting PR to be included upstream) has these features.

package main

import (
"time"

"github.com/Scalingo/go-workers"
)

func main() {
workers.Configure(map[string]string{
"process": "client1", "password": "secret",
"server": "localhost:6379", "namespace": "goworkers",
})
workers.EnqueueIn("myqueue", "MyRubyWorker", 2*60, []string{"in two minutes"})

now := time.Now()
hoursTo9 := (time.Duration(9 - now.Hour())) * time.Hour
at := now.AddDate(0, 0, 1).Truncate(time.Hour).Add(hoursTo9).Unix()

workers.EnqueueAt("myqueue", "MyGoWorker", at, []string{"tomorrow at 9"})
}

Instead of using workers.Enqueue, you can now use workers.EnqueueIn or workers.EnqueueAt (the same way you would use enqueue_in or enqueue_at with Sidekiq. The previous example uses those functions, and calculates the date of tomorrow 9:00am.

Conclusion

With go-workers, it’s really easy to interface your jobs between go and ruby to enjoy the best of both worlds. For instance: nice front-end gems for HTML/Email rendering for ruby and efficient concurrent tasks handling for Go.

Code samples: https://github.com/Scalingo/go-workers-examples Go worker library: https://github.com/Scalingo/go-workers Sidekiq: http://sidekiq.org Homepage: https://appsdeck.eu

Credits

Gopher image by Renee French (Creative Commons Attributions 3.0) Sidekiq logo © 2014 Contributed Systems LLC, contribsys.com

– Léo Unbekandt, CTO @ Appsdeck