Download Multiple Files from Soracom Harvest with API/CLI
Written by: Neel Chopra
Published: July 30, 2021
Being able to upload, download, and manage files associated with your IoT devices are some of the key features we introduced with Soracom Harvest back in 2019.
When a file is uploaded to Soracom Harvest, it is stored automatically for up to 2 years at no extra cost. This gives developers ample time to refine their applications without worrying about storage fees.
While the User Console gives simple access to uploaded files, going through them one by one can become cumbersome. You could be collecting a large number of files as your fleet grows.
That’s why we’re excited to announce that you can download multiple files at once via API or CLI. This provides easier access to your files if you need them for longer than two years.
In this article:
- What is Soracom Harvest?
- Download Multiple Files with API/CLI
- Use Cases for Stored Files
- File Storage Retention Period
- How to Confirm File Expiration Dates
1. What is Soracom Harvest?
Soracom Harvest is a service that collects, visualizes, and stores large files from IoT devices. You can also upload files to Harvest from the User Console and access them from your IoT device.
2. How to Download Multiple Files with API/CLI
In order to simplify the process of obtaining your stored files, we have now added the ability to download multiple files at once through the Soracom API and CLI.
The basic process for downloading multiple files looks like this:
- Use the FileEntry:listFiles to get a list of all of the files currently stored in Harvest. This API will return up to 100 files at a time, so if you currently have more than 100 files, you will need to repeat this API a few times to get the full list.
- Once you have the full list of files, simply loop through the list using the FileEntry:getFile API in order to automatically download each individual file in the list.
Depending on how your files are organized in Harvest or which files you want to download, you can add an extra step in between to sort by date, file size, or other parameters based on your requirements. Then, when you’re ready to download, just let the loop in the second step take care of all of your files in one go!
To help you get started, we have prepared a sample Python script. This script adds a few extra features, such as authenticating the access to the Soracom API using an AuthKey, and providing some useful parameters for how you want the files to be downloaded. Be sure to take a look at the readme file for more instructions on how to set up and run the script.
If you are using the Soracom CLI tool, you can also perform the same steps with the soracom files list and soracom files get commands. Compared to the API, Soracom CLI will take care of authentication automatically, so there is no need to worry about setting up authentication separately.
Read here for more information.
3. Use Cases for Stored Files
There are a number of different use cases where developers will benefit from downloading and uploading files to Harvest.
Two common scenarios we have seen for leveraging have been::
1. Uploading Files from Your IoT Devices
You can manage large files by uploading them directly from your device to Harvest. For example, if your device has a camera, you can program it to upload the captured images and logs to Harvest.
You can also create interactive dashboards in Soracom Lagoon where you can view the captured images together with corresponding sensor data.
2. Downloading Files to Your IoT Devices
Developers can use stored files for periodic maintenance of devices. For example, you can upload device firmware updates to Harvest from the User Console. Then, you can program your device to periodically check Harvest for new firmware updates, download, and install them.
4. File Storage Retention Period
While File Storage in Harvest provides many benefits, there are a few limitations to be aware of.
By default, Harvest will store your files for up to 2 years from their initial upload date and automatically delete them afterward.
With the release of the feature of storing files now approaching 2 years of service, files uploaded shortly after the service launched will be scheduled for automatic deletion soon.
If you were an early adopter of uploading files to Harvest, you may want to backup your earliest uploads, as they will likely be deleted soon.
5. How to Confirm File Expiration Dates
You can check the expiration date of your files from the User Console.
Click on Menu, select Data Storage and visualization, and then click on SORACOM Harvest Files.
Navigate to your file(s) and check the Expiry Time to confirm when they will be deleted.
Summary
This new ability to download multiple files from Soracom Harvest using the Soracom API and CLI enables users to manage their files easier and save their files before they are deleted.
Beyond just downloading a copy of your files before they expire, you can also implement the same API and CLI directly into your application to automate your file management operations.
To learn more about Soracom Harvest, visit our developer docs.
Any questions? Contact us here.
