Background on Azure Functions
- Azure Functions are a serverless offering on the Microsoft Azure platform. Serverless meaning that you don’t need to manage/worry about what server the functions actually run on.
- If I understand it right, when you setup a new Azure Functions instance, it creates an docker container for your functions inside of your tenant docker container that runs on host. Your functions in there all can sort of interact with each other (to your joy or detriment). You can even write to the filesystem if you want to enable that.
- Azure Functions sometimes stop running mid process. It could be a good idea for your code to be able somewhat idempotent, meaning, if your function gets run twice for whatever reason with the same input, you should be able to detect that and not do anything the 2nd time.
Building blocks
These are some of the building blocks that I started learning about that may be of great interest to you:
API Management Gateway
The Consumption plan of this is free for like 2 million calls per month. The Basic and above plans cost at least like $150/mo but come with a developer portal instance that you can use to host documentation of your APIs. Think like a swagger or postman interactive documentation page. And more if you want to.
Each API in the API gateway can point to a different backend host. Makes it very easy to have a single front that ingests your data!
The API gateway can track each request to it as it heads to the backend server and gets responded to. Any errors that the backend server throws show up in the API logs. The timings of requests is also tracked so that you can do some profiling of what is taking how long.
Azure Functions
MS Recommends that you keep the dev/uat/production versions of your functions in separate azure functions instances so that their load doesn’t impact each other. If you don’t want separate function instances, you can create a 2nd (or 3rd if you get azure premium function) slot and use env variables for your function to detect which slot its running it.
Azure functions get run through one of many possible triggers see the “trigger” column from this link. HTTP and Timer triggers are extremely common. You can also use output-bindings to SEND data out of your function. SendGrid, blobstorage, storagequeues, etc. VERY NIFTY STUFF.
There are lots of things you can do to tune your functions for powershell. Its worth keeping this PowerShell reference page handy.
Storage accounts
Blob storage (meaning basically a fileshare like an AWS s3 bucket), Storage queue (FIFO messaging), and more. These are for stuff that you don’t want to put into a database. Use the Azure Storage Explorer to have better access to the data in these.
Blob storage containers can be a trigger for Azure Functions, however, they are only rated to trigger at like 100 operations per minute. If you want to be able to handle peak load stuff, don’t use Storage account blob triggers.
Accessing storage accounts from powershell isn’t super easy. I believe that someone can get Manage Service Identities (MSI - Its recently been renamed to something else) to work with these, but I couldn’t. I reverted to use SAS tokens and just querying storage account data by URI with that SAS token.
Databases
Use Azure Data Studio and add in the Projects addin to be able to use a DB project and do schema compare and deploy project to Database. Super awesome to be able to use those tools! We pair with Azure SQL database instances. Works great. Scales nicely. Cheap. We do database security with Azure Identity groups
Security: You don’t want to grant database instance wide permissions for users. You want to grant PER DATABASE permissions to tables. Use external provider for that:
CREATE USER [azureUserOrGroupName] FROM EXTERNAL PROVIDER
GO
GRANT CONNECT TO [azureUserOrGroupName]
GO
GRANT SELECT ON OBJECT::TableNameFoo TO [azureUserOrGroupName]
GO
Powershell standard tools for accessing a database apply here. You can deploy modules to your function, or use the managed dependencies (I couldn’t get them to work). SqlServer or DbaTools modules, or even the Invoke-SqlCmd2 script work just fine.
EventGrid Subscriptions
EventGrid (not to be confused with SendGrid email) subscriptions are the internal messaging/notification system bus that really rock if you need to chain stuff together and do it FAST. They are also a lot to take in sometimes.
Where a blob storage account can only send somewhere around 100 blobs per minute to your azure function, you can use EventGrid (EG) subscription to get notified of thousands of events per minute and then query the blob storage account thousands of times per minute without issue. Its weird, but whatever.
Tying it all together
For us, we needed 2 stage Azure functions: 1 function to respond to external requests and another to actually do the work on the data that gets submitted. The 2nd one took anywhere from 5s-300s, so we didn’t want the caller request to be waiting on that processing to finish before they get a response.
We used API management as the entry point. It calls an internal-only Azure function which basically just validates the data and accepts it into a blob in the storage account (we don’t use a queue item because those are limited to 64k). The function replies to the user and says, “We accepted the data (status 201 or something).”
The blob getting created triggers an eventGrid message that triggers our 2nd stage azure function which goes and sees if the blob (meaning file) still needs processed according to the database. If yes, it gets the contents of the blob and spends a few minutes processing it and creating a response in the database and possibly to another blob storage container.
Now that it is done processing, the next time the caller asks about the status of that request, it’ll be able to get the results that are sitting in the database.
Further reading
I haven’t mentioned (or used) Durable Functions, which is a way to bring stateful-ness into your serverless, stateless functions. Basically, when you need to run several functions, Azure can checkpoint the output of each one once it completes and when they are all ready, send them on down the line to the thing that expects everything. Here is the overview from MS.
Discussion
Discuss on reddit