Yorc Plugins (Advanced)¶

Yorc exposes several extension points. This section covers the different ways to extend Yorc and how to create a plugin.

Note

This is an advanced section! If you are looking for information on how to use Yorc or on existing plugins please refer to our main documentation

Yorc extension points¶

Yorc offers several extension points. The way to provide extensions is to load plugins within Yorc. Those plugins could be use to enrich Yorc or to override Yorc builtin implementations. If there is an overlap between a Yorc builtin implementation and an implementation provided by a plugin the plugin implementation will be preferred.

TOSCA Definitions¶

One way to extend Yorc is to provide some TOSCA definitions. Those definitions could be used directly within deployed applications by importing them using the diamond syntax without providing them into the CSAR:

imports:
  - this/is/a/standard/import/to/an/existing/file/within/csar.yaml
  - <my-custom-definition.yaml>
  - <normative-types.yml>

Both two latest imports are taken directly from Yorc not within the CSAR archive.

Delegate Executors¶

Some nodes lifecycle could be delegated to Yorc. In this case a workflow does not contain set_state and call_operation activities. Their workflow is considered as “delegate” and acts as a black-box between the initial and started state in the install workflow and the started to deleted states in the uninstall workflow.

Those kind of executors are typically designed to handle infrastructure types.

You can extend Yorc by adding new implementations that will handle delegate operations for selected TOSCA types. For those extensions you match a regular expression on the TOSCA type name to a delegate operation. For instance you can match all delegate operations on type named my\.custom\.azure\..*.

Operation Executors¶

Those kind of executors handle call_operation activities.

In TOSCA operations are defined by their “implementation artifact”. With a plugin you can register an operation executor for any implementation artifact.

Those executors are typically designed to handle new configuration managers like chef or puppet for instance.

Action Operators¶

Action Operators handle the execution of asynchronous operations. These executors are typically designed to handle the monitoring of a job state.

Infrastructure Usage Collector¶

An infrastructure usage collector allows to retrieve information on underlying infrastructures like quota usage or cluster load.

You can register a collector for several infrastructures.

How to create a Yorc plugin¶

Yorc supports a plugin model, plugins are distributed as Go binaries. Although technically possible to write a plugin in another language, plugin written in Go are the only implementations officially supported and tested. For more information on installing and configuring Go, please visit the Golang installation guide. Yorc and plugins require at least Go 1.11.

This sections assumes familiarity with Golang and basic programming concepts.

Initial setup and dependencies¶

Starting with Yorc 3.2 the official way to handle dependencies is Go modules. This guide will use Go modules to handle dependencies and we recommend to do the same with your plugin.

# You can store your code anywhere but if you store it into your GOPATH you need the following line
$ export GO111MODULE=on
$ mkdir my-custom-plugin ; cd my-custom-plugin
$ go mod init github.com/my/custom-plugin
go: creating new go.mod: module github.com/my/custom-plugin
$ go get -m github.com/ystia/yorc/v4@v4.0.0-M1
$ touch main.go

Building the plugin¶

Go requires a main.go file, which is the default executable when the binary is built. Since Yorc plugins are distributed as Go binaries, it is important to define this entry-point with the following code:

package main

import (
  "github.com/ystia/yorc/v4/plugin"
)

func main() {
  plugin.Serve(&plugin.ServeOpts{})
}

This establishes the main function to produce a valid, executable Go binary. The contents of the main function consumes Yorc’s plugin library. This library deals with all the communication between Yorc and the plugin.

Next, build the plugin using the Go toolchain:

$ go build -o my-custom-plugin

To verify things are working correctly, execute the binary just created:

$ ./my-custom-plugin
This binary is a plugin. These are not meant to be executed directly.
Please execute the program that consumes these plugins, which will
load any plugins automatically

Load custom TOSCA definitions¶

You can instruct Yorc to make available some TOSCA definitions as builtin into Yorc. To do so you need to get the definition content using the way you want. For simplicity we will use a simple go string variable in the below example. Then you need to update ServeOpts in your main function.

package main

import (
  "github.com/ystia/yorc/v4/plugin"
)

var def = []byte(`tosca_definitions_version: yorc_tosca_simple_yaml_1_0

metadata:
  template_name: yorc-my-types
  template_author: Yorc
  template_version: 1.0.0

imports:
  - <normative-types.yml>

artifact_types:
  mytosca.artifacts.Implementation.MyImplementation:
    derived_from: tosca.artifacts.Implementation
    description: My dummy implementation artifact
    file_ext: [ "myext" ]

node_types:
  mytosca.types.Compute:
    derived_from: tosca.nodes.Compute

`)

func main() {
  plugin.Serve(&plugin.ServeOpts{
    Definitions: map[string][]byte{
      "mycustom-types.yml": def,
    },
  })
}

Implement a delegate executor¶

Now we will implement a basic delegate executor, create a file delegate.go and edit it with following content.

package main

import (
  "context"
  "log"

  "github.com/ystia/yorc/v4/config"
  "github.com/ystia/yorc/v4/deployments"
  "github.com/ystia/yorc/v4/events"
  "github.com/ystia/yorc/v4/locations"
  "github.com/ystia/yorc/v4/tasks"
  "github.com/ystia/yorc/v4/tosca"
)

type delegateExecutor struct{}

func (de *delegateExecutor) ExecDelegate(ctx context.Context, conf config.Configuration, taskID, deploymentID, nodeName, delegateOperation string) error {

  // Here is how to retrieve location config parameters from Yorc
  locationMgr, err := locations.GetManager(conf)
  if err != nil {
    return err
  }
  locationProps, err := locationMgr.GetLocationPropertiesForNode(deploymentID, nodeName, "my-plugin-location-type")
  if err != nil {
    return err
  }
  for _, k := locationProps.Keys() {
      log.Printf("configuration key: %s", k)
  }
  log.Printf("Secret key: %q", locationProps.GetStringOrDefault("test", "not found!"))

  // Get a consul client to interact with the deployment API
  cc, err := conf.GetConsulClient()
  if err != nil {
    return err
  }
  kv:= cc.KV()

  // Get node instances related to this task (may be a subset of all instances for a scaling operation for instance)
  instances, err := tasks.GetInstances(kv, taskID, deploymentID, nodeName)
  if err != nil {
    return err
  }

  // Emit events and logs on instance status change
  for _, instanceName := range instances {
    deployments.SetInstanceStateWithContextualLogs(ctx, kv, deploymentID, nodeName, instanceName, tosca.NodeStateCreating)
  }

  // Use the deployments api to get info about the node to provision
  nodeType, err := deployments.GetNodeType(cc.KV(), deploymentID, nodeName)

  // Emit a log or an event
  events.WithContextOptionalFields(ctx).NewLogEntry(events.LogLevelINFO, deploymentID).Registerf("Provisioning node %q of type %q", nodeName, nodeType)

  for _, instanceName := range instances {
    deployments.SetInstanceStateWithContextualLogs(ctx, kv, deploymentID, nodeName, instanceName, tosca.NodeStateStarted)
  }
  return nil
}

Now you should instruct the plugin system that a new executor is available and which types it supports. This could be done by altering again ServeOpts in your main function.

package main

import (
  "github.com/ystia/yorc/v4/plugin"
  "github.com/ystia/yorc/v4/prov"
)

// ... omitted for brevity ...

func main() {
  plugin.Serve(&plugin.ServeOpts{
    Definitions: map[string][]byte{
      "mycustom-types.yml": def,
    },
    DelegateSupportedTypes: []string{`mytosca\.types\..*`},
    DelegateFunc: func() prov.DelegateExecutor {
      return new(delegateExecutor)
    },
  })
}

Implement an operation executor¶

An operation executor could be implemented exactly in the same way than a delegate executor, except that it need to support two different functions, ExecOperation and ExecOperationAsync. The first one is the more common use case while the latest is designed to handle asynchronous (non-blocking for long running) operations, like jobs execution typically. In this guide we will focus on ExecOperation please read our documentation about jobs for more details on asynchronous operations. You can create a operation.go file with following content.

package main

import (
  "context"
  "fmt"
  "time"

  "github.com/ystia/yorc/v4/config"
  "github.com/ystia/yorc/v4/events"
  "github.com/ystia/yorc/v4/prov"
)

type operationExecutor struct{}

func (oe *operationExecutor) ExecAsyncOperation(ctx context.Context, conf config.Configuration, taskID, deploymentID, nodeName string, operation prov.Operation, stepName string) (*prov.Action, time.Duration, error) {
  return nil, 0, fmt.Errorf("asynchronous operations %v not yet supported by this sample", operation)
}

func (oe *operationExecutor) ExecOperation(ctx context.Context, cfg config.Configuration, taskID, deploymentID, nodeName string, operation prov.Operation) error {
  events.WithContextOptionalFields(ctx).NewLogEntry(events.LogLevelINFO, deploymentID).RegisterAsString("Hello from my OperationExecutor")
  // Your business logic goes there
  return nil
}

Then you should instruct the plugin system that a new executor is available and which implementation artifacts it supports. Again, this could be done by altering ServeOpts in your main function.

// ... omitted for brevity ...

func main() {
  plugin.Serve(&plugin.ServeOpts{
    Definitions: map[string][]byte{
      "mycustom-types.yml": def,
    },
    DelegateSupportedTypes: []string{`mytosca\.types\..*`},
    DelegateFunc: func() prov.DelegateExecutor {
      return new(delegateExecutor)
    },
    OperationSupportedArtifactTypes: []string{"mytosca.artifacts.Implementation.MyImplementation"},
    OperationFunc: func() prov.OperationExecutor {
      return new(operationExecutor)
    },
  })
}

Logging¶

Using the log standard library or Yorc log module github.com/ystia/yorc/v4/log in plugin code, log data from the plugin will be automatically sent to the Yorc Server parent process. Yorc will parse these plugin logs to infer their log level and filter them if these are debug messages and debug logging is disabled. It will then display messages, prefixed by the plugin name and suffixed by the timestamp of their creation on the plugin.

Plugin log messages levels are inferred this way by Yorc Server :

A message sent by the plugin using Yorc log module github.com/ystia/yorc/v4/log function log.Debug(), log.Debugf() or log.Debugln() will have the level DEBUG, all other messages will have the level INFO on Yorc server.

A message sent by the plugin using the log standard library will have the level INFO, except if this message is prefixed by one of these values: [DEBUG], [INFO], [WARN], [ERROR], in which case the message will have the log level corresponding to this value on Yorc server.

See an example in next section of a plugin logs with debug logging enabled on Yorc server.

Using Your Plugin¶

First your plugin should be dropped into Yorc’s plugins directory before starting Yorc. Yorc’s plugins directory is configurable but by default it’s a directory named plugins in the current directory when Yorc is launched.

By exporting an environment variable YORC_LOG=1 before running Yorc, plugin debug logs will be displayed, else these debug logs will be filtered and other plugin logs will be displayed, as described in previous section.

# Run consul in a terminal
$ consul agent -dev
# Run Yorc in another terminal
$ mkdir plugins
$ cp my-custom-plugin plugins/
$ YORC_LOG=1 yorc server
...
2019/02/12 14:28:23 [DEBUG] Loading plugin "/tmp/yorc/plugins/my-custom-plugin"...
2019/02/12 14:28:23 [INFO]  30 workers started
2019/02/12 14:28:23 [DEBUG] plugin: starting plugin: /tmp/yorc/plugins/my-custom-plugin []string{"/tmp/yorc/plugins/my-custom-plugin"}
2019/02/12 14:28:23 [DEBUG] plugin: waiting for RPC address for: /tmp/yorc/plugins/my-custom-plugin
2019/02/12 14:28:23 [DEBUG] plugin: my-custom-plugin: 2019/02/12 14:28:23 [DEBUG] plugin: plugin address: unix /tmp/plugin262069315 timestamp=2019-02-12T14:28:23.499Z
2019/02/12 14:28:23 [DEBUG] plugin: my-custom-plugin: 2019/02/12 14:28:23 [DEBUG] Consul Publisher created with a maximum of 500 parallel routines. timestamp=2019-02-12T14:28:23.499Z
2019/02/12 14:28:23 [DEBUG] Registering supported node types [mytosca\.types\..*] into registry for plugin "my-custom-plugin"
2019/02/12 14:28:23 [DEBUG] Registering supported implementation artifact types [mytosca.artifacts.Implementation.MyImplementation] into registry for plugin "my-custom-plugin"
2019/02/12 14:28:23 [DEBUG] Registering TOSCA definition "mycustom-types.yml" into registry for plugin "my-custom-plugin"
2019/02/12 14:28:23 [INFO]  Plugin "my-custom-plugin" successfully loaded
2019/02/12 14:28:23 [INFO]  Starting HTTPServer on address [::]:8800
...

Now you can create a dummy TOSCA application topology.yaml

tosca_definitions_version: alien_dsl_2_0_0

metadata:
  template_name: TestPlugins
  template_version: 0.1.0-SNAPSHOT
  template_author: admin

imports:
  - <mycustom-types.yml>

node_types:
  my.types.Soft:
    derived_from: tosca.nodes.SoftwareComponent
    interfaces:
      Standard:
        create: dothis.myext

topology_template:
  node_templates:
    Compute:
      type: mytosca.types.Compute
      capabilities:
        endpoint:
          properties:
            protocol: tcp
            initiator: source
            secure: true
            network_name: PRIVATE
        scalable:
          properties:
            max_instances: 5
            min_instances: 1
            default_instances: 2

    Soft:
      type: my.types.Soft

  workflows:
    install:
      steps:
        Compute_install:
          target: Compute
          activities:
            - delegate: install
          on_success:
            - Soft_creating
        Soft_creating:
          target: Soft
          activities:
            - set_state: creating
          on_success:
            - create_Soft
        create_Soft:
          target: Soft
          activities:
            - call_operation: Standard.create
          on_success:
            - Soft_created
        Soft_created:
          target: Soft
          activities:
            - set_state: created
          on_success:
            - Soft_started
        Soft_started:
          target: Soft
          activities:
            - set_state: started
    uninstall:
      steps:
        Soft_deleted:
          target: Soft
          activities:
            - set_state: deleted
          on_success:
            - Compute_uninstall
        Compute_uninstall:
          target: Compute
          activities:
            - delegate: uninstall

Finally you can deploy your application and see (among others) the following logs:

$ yorc d deploy -l --id my-app topology.yaml
<...>
[2019-02-12T16:51:55.207420877+01:00][INFO][my-app][install][5a7638e8-dde2-48e7-9e5a-89350ccd99a7][8f4b31da-8f27-456e-8c25-0520366bda30-0][Compute][0][delegate][install][]Status for node "Compute", instance "0" changed to "creating"
[2019-02-12T16:51:55.20966624+01:00][INFO][my-app][install][5a7638e8-dde2-48e7-9e5a-89350ccd99a7][8f4b31da-8f27-456e-8c25-0520366bda30-1][Compute][1][delegate][install][]Status for node "Compute", instance "1" changed to "creating"
[2019-02-12T16:51:55.211403476+01:00][INFO][my-app][install][5a7638e8-dde2-48e7-9e5a-89350ccd99a7][8f4b31da-8f27-456e-8c25-0520366bda30][Compute][][delegate][install][]Provisioning node "Compute" of type "mytosca.types.Compute"
[2019-02-12T16:51:55.213793985+01:00][INFO][my-app][install][5a7638e8-dde2-48e7-9e5a-89350ccd99a7][8f4b31da-8f27-456e-8c25-0520366bda30-0][Compute][0][delegate][install][]Status for node "Compute", instance "0" changed to "started"
[2019-02-12T16:51:55.215991445+01:00][INFO][my-app][install][5a7638e8-dde2-48e7-9e5a-89350ccd99a7][8f4b31da-8f27-456e-8c25-0520366bda30-1][Compute][1][delegate][install][]Status for node "Compute", instance "1" changed to "started"
<...>
[2019-02-12T16:51:55.384726783+01:00][INFO][my-app][install][5a7638e8-dde2-48e7-9e5a-89350ccd99a7][3d640a5a-3093-4c7b-83e4-c67e57a1c430][Soft][][standard][create][]Hello from my OperationExecutor
<...>
[2019-02-12T16:51:55.561607771+01:00][INFO][my-app][install][5a7638e8-dde2-48e7-9e5a-89350ccd99a7][][][][][][]Status for deployment "my-app" changed to "deployed"

Et voilà !