Let’s Talk IoT Devices: Sony’s Spresense

Hello and welcome to the second episode of Soracom’s Let’s Talk IoT Devices webinar series. Today we are going to focus on Sony’s Spresense. My name is Dora Terjek and I am working as an IoT Device Product Manager within Soracom. I have been working within the IoT field for over ten years now, and today we are co hosting this episode with Camilla from Sony. Camilla, would you mind introducing yourself? Of course. Hey, it’s so good to be here with you today, Dora. Hello everyone. My name is Camilla Souza. I’m an engineer here at Sony and I work with the Spresense team. And within the team, my hope is to help build and develop community. So I do things like tutorials and YouTube videos and webinars like this one, And this is all to give the users more resources so they can get started with Spresense. Excellent. Welcome here today. Thank you. Really, really good to have you. Yeah, of course. Before we look at the actual agenda for today, let’s talk about housekeeping. Please raise your questions and add them to the built in webinar chat tool. We are going to be happy to answer you during or after this webinar. And when it comes to today’s session, we broke the webinar into four bigger pieces. We will first talk about Spresense capabilities and what their accessories bring onto the table. Then we will talk about three different development environments and also the support tools that Sony offers behind Spresence, we will bring some very cool use cases, and we are also going to talk about how Soracom can connect a Spresense unit. When it comes to Soracom, not many of you might have heard about us. We are a Japanese company that was founded in twenty fifteen by AWS and telco veterans. We provide connectivity and platform services, and we are really, really proud that we have over twenty thousand customers that vary from different industries, including agriculture, consumer electronics, healthcare, industrial automation and many, many more. We also have over five million devices that are powered by Soracom SIM cards as of today. Our headquarter is based in Tokyo, but we also have offices in London as well as in Seattle. And our global team is spread across the whole world. We have been building a quite strong partnership with Sony and that’s the reason why we are sitting here today. Yes. Why don’t we take a look at Camilla, a short intro video about Spresence? Yeah, let’s do it. A quick introduction to the Spresense microcontroller board. The Spresense microcontroller board from Sony is a unique combination of extensive computing power but with a power consumption low enough to run on a battery. Despite its compact footprint, Spresense is powered by a high performance six core microcontroller with a clock speed of one hundred and fifty six MHz. Spresense features an integrated GPS, high res audio output, multiple microphone inputs, as well as a camera interface for Sony’s own five megapixel CMOS sensor. All of the development tools, documentations, tutorials as well as discussion forums can be found on the Spresense developer website. Developer. Sony dot com slash spresense. What a remarkable set of hardware it is! Now that we have seen some of the major capabilities, would you mind telling us about the main strengths of Spresence? Yes, of course. I really think it’s this unique combination of high computing power, but at the same time low power consumption, that makes it really ideal for IoT scenarios. Also with the possibility of using GPS and high resolution audio, it’s just such a good set of features, you know. Also, it is a good environment, like a good ecosystem where you have the main board, but you also have some extension boards that give it more functionality and more possibilities. I think that’s really cool. Excellent. How does it work when it comes to prototyping versus mass production? What is Spresence more recommended for? I think you can definitely do both. It’s really quick and easy for prototyping, but we have cases of mass production. For example, we have Omnipresent, which is a company based in South Africa, and they are using the main board with the camera board for remote monitoring of cellular towers. When it comes to additional customers, what are the major or target customer cases that Spresense is targeting? So, there is a focus, a main focus on businesses, but the hardware is available for anyone who wants to purchase and try it. We have vast documentation online for end developers who want to start with it, to be really easy. Cool, really good. Let’s look at the actual target applications. Yeah, so because of this unique combination of features, Spresense is really good for applications where you have to do some computation on the edge. So for applications where you need to do some machine learning, data filtering, sensor analysis, as Spresense really shines. And we’ve seen it being used in, for example, for smart cities, for remote monitoring, like we said before, industrial IoT and many, many other applications. Great. How about walking us through on the modular setup of Spresence and what are the optional accessories and the hardware extensions? How do they all come together in Spresence? So in our family we have five boards. So we have the main board, we have two extension boards. One of them is the LTE extension board which adds connectivity to the bundle. We have two camera boards also and each one has slightly different purposes. And we are also compatible with Arduino Uno, so you have many Arduino shields in the market, you can have add ons, so there are lots of possibilities. Nice. Can you tell us a little bit more about the main board, which is the actual brain of Spresence? Yeah, of course. So this is the main board. It has our processor on it, also has GPIO pins, it has one connector for the camera, it has the GNSS antenna, and like you said, this is the brains of the operation, our processor sits. And this is connecting to the actual extension board, right? Yes, and they connect through this board to board connector here. You see you have one on the extension board, you also have one on the LTE board. Nice. So they pair easily like this. And the camera boards, they actually connect through a flat cable. They come in the packaging, we just don’t have it here. But yeah, they connect in this flat cable connector. It’s going to look like something similar, Yes, exactly, exactly perfect. We have a perfect example right here. This is a regular camera board and it connects to the main board and this one is LTE. Cool. And when it comes to actual connectivity, you mentioned that the SIM card would go on this LTE board that supports both narrowband IoT and LTE M? Yeah, exactly, it supports both. Here we have an antenna, an onboard antenna. We have the card holder for the SIM card. We also have an SD card holder. Here we have some microphone pins, we have a headphone jack. We have a little jumper to choose the voltage. Awesome. Also, extension board has some of the same features. It has the headphone jack, it has the SD card holder and the microphone pins too. And you mentioned earlier the two different camera types that Spresense supports. What is the major difference between those two? Yeah, of course. So, this is the main camera board and this is the HDR. So, at first you see that they have a little bit of a size difference. Also, the lens on the HDR camera is bigger and the focus is adjustable manually. So you can focus on objects that are closer to the camera. When you look at pixel counts alone, the regular camera board has a higher resolution. So if your application is mainly focused on the resolution, then the camera board is the one for you. But the HDR camera board is really good in specific scenarios. So for example, if you have an environment that has very low lighting conditions, then the regular camera would present a very dark picture. But with the HDR camera board, you have a better, much better visual results. You can see the objects on the picture much more clear. Also, if you have, for example, an object with a backdrop of very strong lighting, that can also be a situation where the HDR camera is going to give you much better visibility. And we have comparative pictures in our website for the users that want to go and see the difference. Okay. And last but not least, you already brought up Arduino Uno, and we know that its presence is also supporting some of the main shields of Arduino. Can you tell us a bit more about that, especially what developers should think about before selecting one of the shields or some of the shields? Yes, of course. Like I said, this is an Ethernet shield, This one connects with the extension board. It pairs up really easily like this. And this is just one example of shields in the market that are compatible. There are many others. There are also add on boards that go with sorry, with the main board. One thing that you should pay attention is that there are differences between Spresence and the Arduino Uno. On the website we have all these differences listed, but as an example, the main board, the voltage is one point eight, the logical voltage is one point eight, some of the add ons on the market have three point three or maybe five, so that is one thing that users should take a look at before buying. Right, cool. Yeah. It is actually quite rare to find a developer board that has so high performance, processing performance, with so low power. So I think Spresense is a very, very optimal developer tool for many applications. Can you actually walk us through, Camilla, the development flow and the development process? Yeah, of course. I think the development flow is going to be dependent on which environment you choose to develop with. We have three options and we can go into a bit more detail about them later. But what is common about those three is that first you have to see if you have an operating system that we support. So we support Windows, Mac, and Linux starting in different versions. So that’s something you can check out in our Getting Started guide. And then the first step is to do the Spresence setup. And again, that’s going to differ a little bit depending on which option you choose to go with. But after this setup, you create your project and we have many examples in our website to help and to give inspiration. And then, mostly, you build it and you flash it to the board and that’s like the main flow. Brilliant. Yeah. Great. And you mentioned that there are three different development environments. Can you tell us which three are those? Yeah, of course. So we have the Sony’s original Spresence SDK, which is based on NuttX. We also have the possibility to use the Arduino IDE, which is an environment that is very well known by many developers. And we also have the possibility to use CircuitPython, which is a programming language that is based on Python and that we have ported into our environment. And so to go into a little bit more details on what are the differences between them and in which situation you would use them, With the Spresence SDK, you have full access to all of Spresence functionalities. So it gives the user much more control. There are many things that you can program and many configurations that you can choose from in your SDK. For example, if you have an application that you really need to have very, very low power, there are low power modes that you can choose from and there are many other configurations that you choose from. And with the SDK, you can do all that. Okay. But for the control it adds a little bit of complexity, right? And in the Arduino IDE, we have an environment that is a little bit easier to start with, especially for people that are already kind of familiar with the environment and you can import the Spresense library. And what happens in the Spresense library is that all these configurations and these choices that I talked about, we pre choose it, we pre configure it to make it as generic as possible and then we embed it into the Arduino IDE. So this is sort of the trade off, right? You have complexity and control. And with the CircuitPython, for users that want to use, are familiar with the Python language, but the support is still ongoing. There are some features that are still not supported. I see. I see, I see. Earlier we touched base about the computing capabilities, and we all know that machine learning is becoming more and more relevant within IoT applications. So, when it comes to creating machine learning applications, what type of tools does Spresense offer to developers? Yeah, so one platform that’s becoming really popular with our users is Edge Impulse. So Edge Impulse is a leading machine learning platform for microcontrollers and they have a graphical interface that helps you go through all of the steps in your machine learning process. So you can acquire data, you can create your own data set, you can train your model, test it, and then deploy this trained model onto the Spresense. And this all with the help of a good user interface. Also we have support for TensorFlow. TensorFlow is the open source Google library for machine learning. And they have TensorFlow Lite, which we have support for. And also there is NNabla, which is Sony’s neural network library for neural networks. So those are the tools that users can use. Quite a few options to pick from. We all know Spresence is an open source microcontroller board and you already brought up plenty of examples of tutorials and different development information and data that can be found on the website of Sony. Can you tell us a bit more about what other content can be found on the Developer Works section? Yes, of course. So, in there we have an introduction to the whole environment. We also have getting started guides for all of the development options that we have. We have documentation like examples for again all of the options, API references. We also have a session on troubleshooting that we go through some of the most common problems that we see users facing. There’s also a session on frequently asked questions. Again, like most of the initial questions, they tend to repeat, right? Because they mostly go through the same process, the same learning process. So that would be like the first resource when you have a question or a problem. And then we have the Spresense forum, which we have engineers from Sony looking and trying to answer questions and help the users. But they are also helping each other and growing the community. So it’s a really good place to get support for your project. And you can also add your feedback from time to time on the developer And then we have other types of resources. So for example, we have webinars, we have YouTube videos. So we are trying to populate our YouTube channel with shorter videos that are focused on a specific topic. For example, we have a Getting Started video for the SDK. So you have the option of going through that session both written and also with the video. And yeah, many more. And I encourage everyone that is watching to go check it out. Excellent, great. Earlier we touched base on a few target applications. What is your personal favorite use case or use cases for Spresense? Oh yeah, that’s a really, really cool question to talk about. So recently we had something, we had a contest with Hackster that was so much fun to be a part of. I was one of the judges and it was really cool to see the users thinking about their projects and asking questions and developing these really cool ideas, right? And for the winners we gave Sony prizes, so that was really fun. And I can talk about the first places. We have three winners because I think they’re really, really good projects. So for example, we had one of them developed a they wanted to assist the new driver to not make recurring mistakes, right, basic mistakes. So they put the Spresense board with the camera board on the roof of the car, looking at the driver, and it would flag if the driver made some of the mistakes that were considered as dangerous. Okay. So for example, if they picked up their phone, then it would flag, Very good. Yeah, that’s such a cool application I thought. It’s like so many accidents are caused because people are looking at their phones, right? Also if they would accelerate too hard on a turn or if they would go through a sequence of potholes. So they just thought about these common dangerous behaviors and they trained the model to realize when these behaviors were happening. Also, they had someone who did a classifier of dental casts. So we know that dentists are using these three d printed casts to see problems. And so this user, he did his own data set, he printed a lot of dental casts and he did his classifier to know if the three d printing occurred according to the plan. So if there were any mistakes in the production of these dental casts, it would be flagged. So of course it was a proof of concept, but I thought it was a really interesting and unique idea, right? Yeah, it’s another great example of machine Yeah, and he also used Edge Impulse to build his data set. And also the other winner was really interesting too. They had a bionic arm and they wanted to control it with their own movements. So they had a MyoBand in their arms and it would see the electrical impulses that the muscle. And they would make a gesture and the model would predict what gesture they were trying to make and then the bionic arm would mimic. Very impressive. So yeah, was really, really interesting. Yeah, was such a fun contest to be a part of. It was so hard to because there were so, so many good ideas in there and yeah, it was hard. And then we have some other use cases that are really cool, not necessarily associated with this contest. We have the ambassadors for Edge Impulse for example. One of them was in the agricultural use case. So they had a drone that would hover over the crops. Okay. And when they would identify there was something wrong with the crop, they would send not the picture to be processed elsewhere, they would send just the Oh hey, something is wrong and here is the location. So I thought that was really good, you know, because you don’t want to, that’s the beauty of Spresense, to be able to do some edge computing. Because you don’t want to send too much data, that could be costly. So they would do the processing on the edge and just send back, hey, there’s something wrong with this crop, here’s where it is. And then you can make an educated decision. What else? Then we had a smart parking space and there was a camera looking at a car being parked and depending on where the wheels were, like what angle the wheels were, they would do instructions like steer right, steer left, accelerate to have someone park. And also there was one that would predict when it was time to harvest the crops. So that is important so the producers can be better organized and predict sales. So yeah, there’s many good user cases with Presence. Very lucky. And a lot of these use cases, they need to use cellular connectivity, right? And this is where we partner. Exactly. Yes. That’s where Soracom comes in to help us. So, do you mind talking a little bit about that and what are the options for the developers? Yes, for sure. So, as we were saying earlier, within Soracom, we actually offer different connectivity services. We have a cloud based platform that can handle multiple types of SIM cards and multiple types of connectivity technologies, including low power wide area and LTE M. And as we were saying earlier, this extension board is using a nano SIM card. You can see a Soracom SIM inserted in here. So, this would power up this extension board actually. We have a growing LTE-M footprint worldwide. I think many of you know that LTE-M is a relatively new technology and more and more operators are actually enabling their 4G towers with this new technology. So, we have a growing footprint and feel free to come to our website and check the actual coverage map for LTE M or CAT M1. And I also would like to bring your attention to our connectivity kit, which includes the main board, as well as the LTE extension board, as well as the Soracom SIM card that is going to provide a coupon for the platform services itself. So, you can use the actual coupon for firing up the device. And with this cool combination, we hope we are going to enable many of the developers to create exciting new use cases as well. That’s so great! And what we also would like to show you is where to connect with us. So, we listed a number of different forums and contact us pages, so in case you have technical questions, you can turn to Sony’s Spresense Forum. Then we also have the Contact Us pages for Sony, and in case you have any inquiries about connectivity or SIM cards, we are also there for you from Soracom. I also would like to show you one additional page that’s about our own Soracom blogs and internal content. If you would like to follow us on the Let s Talk IoT Devices series, there is a way to actually subscribe on our social channel for it. And I think with this we are going to be wrapping up for today. Perfect. We have a motto within Soracom that says, You create and we connect. So we hope we are going to be able to help you get that connectivity via SPRESENCE and Soracom in the future. That’s perfect. Thank you for your attention. If you have any questions left that you haven’t answered, please post them in the channel, and we are here to answer them. If we cannot come back to you during this session and answer those questions today, we are going to reach out to you and we are also going to send out the script and the deck to all participants. Cool, yeah. Thank you for your attention for today. Thank you. It was so nice to do this with you. Yes, I agree. Thank you, Camilla. Of course.