Upload data to blob storage with Azure Functions
How to store data sent to an Azure Function in blob storage.
Background
I recently finished moving our Innovasys Luminitix (Link Removed) server from Amazon's EC2 into a hybrid of an Azure Web Role and Azure Blob Storage, and a local machine. It's been running for a couple of weeks now without a spot of bother - I really like Azure.
However, one thing I did need was a way of pulling down the files in blob storage so the local loader service could process them. I ended up writing a rough and ready scheduling app to download any new files (and as an afterthought, modified files or optionally uploading of local files).
tl;dr
Grab the sample from the link at the end of the post!
About the application
The sample project attached to this post is a small application which sits in the system tray without a default user interface. At regular intervals the application connects to the configured Azure storage account sand pulls down any new or modified files, and also optionally uploads files as well.
I wrote this app in a hurry and so I neglected to add incidental things such as logging. If a job fails, it'll display a balloon tip with the error message... but you'll have to catch it fast!
Note: This sample makes use of NuGet packages. If you have version 2.7 or later installed, or enable Package Restore, this should automatically download the missing packages as I'm not including several MB of binary files in a source code download!
I'm not going to describe the source as using the Azure storage API is pretty much as easy as it gets. If you find this example project useful, then I'm glad it helped!
Configuration
Double clicking the system tray icon allows you to configure the storage accounts to look it. I've tested it downloading from two accounts, and downloading + uploading to a further account, and so far it's been running smoothly, but it's rough code, so while it works for me, it might not for you.
Why use a custom scheduler instead of using Windows?
The Windows Task Scheduler is a powerful beast (certainly in latter versions of Windows) and includes many bells and whistles. You could certainly use this to run a download task every few minutes if required.
The main reason I went with using a custom scheduler is because I wanted the system tray icon to let me know what's happening. And if the program exits as soon as it's finished, then I'll not be able to see any errors without having to manually check logs. Which sort of defeats the purpose of using automation.
If you are happy with the stability of the program (or perhaps add the logging that I neglected or some other form of notifications) then you could just change the program to remove the scheduling aspects, and then run it every so often via Windows and save yourself some resources.
A note on the task scheduler
I haven't included the source for the task scheduler, as it's not entirely my own work - the task scheduler I normally use is too tightly integrated with one of our products for me to easily pull it out, and so I chopped together one based partly on this, and partly one some random source I found on the internet. I'll update this download with the full scheduler source once I've properly separated our custom one and remove the plagiarized one.
Limitations
As I mentioned, I built this in a hurry, so it doesn't have a great deal of polish. Amongst the more notable omissions are
- No logging support. It won't tell you what it has downloaded or uploaded
- Sub folder support. It doesn't do sub folders, either local or remote.
- Uploaded files don't keep the local file's modification time stamp. That's actually a real annoyance for me, but unfortunately those properties are read-only for the remote blobs.
Assuming I extend this application to address these (and any other) limitations, I'll update the source download below.
License
The source code associated with this article is licensed under
the MIT License. See license.txt in the download for more
details.
Update History
- 2013-09-08 - First published
- 2020-11-21 - Updated formatting
Like what you're reading? Perhaps you like to buy us a coffee?
Files
Comments
The founder of Cyotek, Richard enjoys creating new blog content for the site. Much more though, he likes to develop programs, and can often found writing reams of code. A long term gamer, he has aspirations in one day creating an epic video game - but until that time comes, he is mostly content with adding new bugs to WebCopy and the other Cyotek products.
Related Articles
Downloading new and changed Azure storage blobs at scheduled intervals
A sample project which downloads new and changed blobs from Azure storage, and optionally uploads new and changed local files.