AzureFunctions

今年もあと少し。ほぼ趣味の範囲を超えないレベルで今年取り組んだテーマの1つにAzure Functions with Pythonがある。あまり情報が無い中、興味本位でサンプルコードを作っては動かして試して得られた情報をシコシコとGithubに上げているうちにナレッジが溜まって来た。それほど多くはないと思うがPythonでAzure Functionsアプリを作りたいという人もいると思うのでノウハウをブログにまとめておく。いきなり水を差すようではあるが、現時点(2017年12月)ではAzure FunctionsのPythonサポータビリティはExperimental(実験的サポート)でありプロダクション向きではない状況であるので、ホントにPythonが好きな人がOn your own riskで楽しんでいただければと思う。

Azure FunctionsのPythonサポート状況

Azure FunctionsのRuntimeには大きく1系と2系の2種類あるが、現時点でPythonは1系でのみExperimentalサポートという状況( See also 言語サポート状況

azure-functions-runtime-version

Experimental(実験的サポート)なので本番での利用は非推奨であり、公式サポートはない(ベストエフォートでのサポートは得られるはず)。また、当然ながらGA言語に比べパフォーマンスは悪い。PythonはFunction呼び出し毎にpython.exeが実行される(GA言語はRuntimeと同じプロセスで実行)。

将来的な話をすると、Azure Functions Runtime 1系でのPythonサポートについては今のExperimentalの域を超えることはないだろう。一方、Runtime 2系ではPythonが正式サポートされるように対応が進められている。ただし時期は未定。この対応については下記Github Issueが切られており、ある程度の対応状況であれば確認可能。Pythonを使う利点の1つに、強力な数理計算、自然言語解析、機械学習系モジュールがあるが、早く安定とパフォーマンスが備わったPythonサーバレスアプリ実行環境でこれら強力なモジュールを活用できたらと思うのは私だけではないだろう。今後の進展に期待。

Hosting Planの選択について

Consumption Plan vs App Service Plan

Azure FunctionsのHosting PlanにはConsumption PlanとApp Service Planの2つがあって、言語に関係なく各プランの特徴は次の通り:

Consumption Plan

App Service Plan

Pythonで使う上で気をつけるポイント

[参考] Coldスタート問題

Python 3.Xランタイムへの変更方法

2017年12月時点のAzure FunctionsのデフォルトPython Runtimeは2.7.8である。Site ExtensionにPython3.5系とPython3.6系が用意されているので、それを利用してFunctionsで利用するPython Runtimeを変更する方法を下記ページに纏めた。

モジュールのインストール方法

pipとKudu DebugConsole/UIを利用した2種類のモジュールインストール方法を下記ページに纏めた。

Pythonサンプルコード

私が試したTriggerとBinding利用サンプルコードは全て下記Githubプロジェクトに追加するようにしている。もしこれを読んでいる皆さんで下記プロジェクトでカバーされていないTrigger/Bindingの組み合わせを試されたら是非ともコントリビュートください。

スライドとHands-onマテリアル

これは今年の11月20日a>と11月28日Azure Antennaで実施したHands-Onセッション(追記:その時の記事)のスライドとHands-Onマテリアルである。よかったらこちらも参考にしていただければと思う。

PythonによるAzureサーバレスアプリケーション開発 / Serverless Application Development with Python from Yoichi Kawasaki

Hands-onマテリアル:

それでは、Enjoy Serverless Application Development with Python!

In this article, I’d like to introduces a solution to collect events from various sources and send them into HTTP Trigger function in Azure Functions using fluent-plugin-azurefunctions. Triggers in Azure Functions are event responses used to trigger your custom code. HTTP Trigger functions allow you to respond to HTTP events sent from fluentd and cook them into whatever you want!


fluent-plugin-azurefunctions

[note] Azure Functions is a (“serverless”) solution for easily running small pieces of code, or “functions,” in Azure. Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data. fluent-plugin-azurefunctions is a fluentd output plugin that enables to collect events into Azure Functions.

Pre-requisites

Setup: Azure Functions (HTTP Trigger Function)

Create a function (HTTP Trigger). First, you need to have an function app that hosts the execution of your functions in Azure if you don’t already have. Once you have an function app, you can create a function. Here are instructions:

A quick-start HTTP trigger function sample is included under examples/function-csharp in Github repository. You simply need to save the code (run.csx) and configuration files (function.json, project.json) in the same Azure function folder. Explaining a little bit about each of files, the function.json file defines the function bindings and other configuration settings. The runtime uses this file to determine the events to monitor and how to pass data into and return data from function execution. The project.json defines packages that the application depends. The run.csx is a core application file where you write your code to process Your jobs. Here is a sample run.csx:

Setup: Fluentd

First of all, install Fluentd. The following shows how to install Fluentd using Ruby gem packger but if you are not using Ruby Gem for the installation, please refer to this installation guide where you can find many other ways to install Fluentd on many platforms.

# install fluentd
sudo gem install fluentd --no-ri --no-rdoc

# create fluent.conf
fluentd --setup <directory-path-to-fluent-conf>

Also, install fluent-plugin-azurefunctions for fluentd aggregator to send collected event data into Azure Functions.

sudo gem install fluent-plugin-azurefunctions

Next, configure fluent.conf, a fluentd configuration file as follows. Please refer to this for fluent-plugin-azurefunctions configuration. The following is a sample configuration where the plugin writes only records that are specified by key_names in incoming event stream out to Azure Functions:

# This is used by event forwarding and the fluent-cat command
<source>
    @type forward
    @id forward_input
</source>

# Send Data to Azure Functions
<match azurefunctions.**>
    @type azurefunctions
    endpoint  AZURE_FUNCTION_ENDPOINT   # ex. https://<accountname>.azurewebsites.net/api/<functionname>
    function_key AZURE_FUNCTION_KEY     # ex. aRVQ7Lj0vzDhY0JBYF8gpxYyEBxLwhO51JSC7X5dZFbTvROs7uNg==
    key_names key1,key2,key3
    add_time_field true
    time_field_name mytime
    time_format %s
    localtime true
    add_tag_field true
    tag_field_name mytag
</match>

[note] If key_names not specified above, all incoming records are posted to Azure Functions (See also this).

Finally, run fluentd with the fluent.conf that you configure above.

fluentd -c ./fluent.conf -vv &

TEST

Let’s check if test events will be sent to Azure Functions that triggers the HTTP function (let’s use the sample function included in Github repo this time). First, generate test events using fluent-cat like this:

echo ' { "key1":"value1", "key2":"value2", "key3":"value3"}' | fluent-cat azurefunctions.msg

As both add_time_field and add_tag_field are enabled, time and tag fields are added to the record that are selected by key_names before posting to Azure Functions, thus actual HTTP Post request body would be like this:

{
    "payload": '{"key1":"value1", "key2":"value2", "key3":"value3", "mytime":"1480195100", "mytag":"azurefunctions.msg"}'
}

If events are sent to the function successfully, a HTTP trigger function handles the events and the following logs can be seen in Azure Functions log stream:

2016-11-26T21:18:55.200 Function started (Id=5392e7ae-3b8e-4f65-9fc1-6ae529cdfe3a)
2016-11-26T21:18:55.200 C# HTTP trigger function to process fluentd output request.
2016-11-26T21:18:55.200 key1=value1
2016-11-26T21:18:55.200 key2=value2
2016-11-26T21:18:55.200 key3=value3
2016-11-26T21:18:55.200 mytime=1480195100
2016-11-26T21:18:55.200 mytag=azurefunctions.msg
2016-11-26T21:18:55.200 Function completed (Success, Id=5392e7ae-3b8e-4f65-9fc1-6ae529cdfe3a)

Advanced Senarios

1. Near Real-time processing

Function Apps can output messages to different means or data stores. For example, fluentd collects events generated from IoT devices and send them to Azure Function, and the the HTTP trigger function transforms the events and processes the data to store in a persistent storage or to pass them to different means. Here are some of options available at the time of writing:

2. Background jobs processing

If the jobs are expected to be large long running ones, it’s recommended that you refactor them into smaller function sets that work together and return fast responses. For example, you can pass the HTTP trigger payload into a queue to be processed by a queue trigger function. Or if the payload is too big to pass into the queue, you can store them onto Azure Blob storage at first, then pass only limited amount of the data into a queue just to trigger background workers to process the actual work. These approaches allow you to do the actual work asynchronously and return an immediate response.

LINKS

END

UPDATED:

Here is a list of fluentd plugins for Microsoft Azure Services.

Plugin NameTarget Azure ServicesNote
fluent-plugin-azurestorageBlob StorageAzure Storate output plugin buffers logs in local file and upload them to Azure Storage periodicall
fluent-plugin-azureeventhubsEvent HubsAzure Event Hubs buffered output plugin for Fluentd. Currently it supports only HTTPS (not AMQP)
fluent-plugin-azuretablesAzure TablesFluent plugin to add event record into Azure Tables Storage
fluent-plugin-azuresearchAzure SearchFluent plugin to add event record into Azure Search
fluent-plugin-documentdbDocumentDBFluent plugin to add event record into Azure DocumentDB
fluent-plugin-azurefunctionsAzure FunctionsAzure Functions (HTTP Trigger) output plugin for Fluentd. The plugin aggregates semi-structured data in real-time and writes the buffered data via HTTPS request to HTTP Trigger Function.
fluent-plugin-azure-loganalyticsLog AnalyticsAzure Log Analytics output plugin for Fluentd. The plugin aggregates semi-structured data in real-time and writes the buffered data via HTTPS request to Azure Log Analytics.

(as of Nov 23, 2016)



fluentd