The beginning of anything is exciting and full of nervous energy. It’s the same when we start working with a new software client. It’s the same when you meet a new friend. It’s the same when you hire a new colleague.

I’m nervous. That’s kind of my consistent state when we start working with a new client. We’ve probably started with over 50 clients since we started 5 years ago. But I still get nervous at the beginning of the engagement, probably for the first month.

Like any relationship, we have to establish trust. New clients don’t trust us yet. They shouldn’t. And the beginning of a new relationship around software development there are always things that come up at the beginning. We’re learning about each other. That takes time.

I know this, but I still get nervous. Luckily we hire good talent that takes care of things. That helps me sleep at night.

So if we’re lucky to start working together, maybe you’ll feel our collective nervous energy over at Augment.

The promise of artificial intelligence is endless. Even Elon Musk is worried that it will take over the world.

We’re pumped for AI and what it can do. But for now, its capabilities are fairly limited. A 2 year old child is well beyond most AI capabilities.

Can you imagine trying to train a computer or robot to walk into a room, see a pile of blocks and know how to create a pyramid of blue blocks? Wowsers. This is nothing for a 2 year old to do. For a robot with AI, it’s state of the art. Many researchers are working on this.

I’ve interviewed a number of robotic professionals and researchers on our podcast: www.flyoverlabs.io.

Steve Cousins, CEO at Savioke – Interview

Julie Shah, Assistant Professor at MIT in the Department of Aeronautics and Astronautics – Interview

Robotics involve both AI and robotic manipulation so it’s much harder than just working with AI and data.

Let’s talk about AI and data. Probably the area where AI has had the most impact is computer vision systems. That problem is perfect for neural networks where each pixel can be easily represented in a neural network system. Computer vision capabilities around static images are pretty amazing.

Computer vision systems can now even label stuff in images. That’s impressive. But compared to human vision systems and maneuvering in our environment, AI computer vision is not good. In very structured environments like a warehouse it can work well.

Check out this podcast with Michtell Weiss, CTO at Seegrid. They have developed a system where forklifts can quickly navigate a manufacturing or warehouse space autonomously.

But it’s a very controlled environment where only so much changes in a day.

Compare that to autonomous cars. Autonomous cars must be able to respond to numerous new circumstances. That’s quite tough. It’s the situations like a kite floating across the street or a kid on a big wheel, that is very tough to account for.

And even training autonomous cars is easier than trying to train a new office manager who might help with 20 different things a day in the office. With autonomous vehicles, you have very few actions: turn the wheel; brake; accelerate. Also signal and some other stuff but the main actions are those three things. An AI can learn by just watching humans driving and adjusting their models accordingly. There is also a ton of programming that needs to happen beyond watching humans drive of course, especially for all of those edge cases like a bright light shining off of a train. I’d rather not run into a train, just saying.

Here’s a podcast on autonomous vehicles we did:

Karl Iagnemma, CEO and Co-Founder of nuTonomy – Interview

AI is awesome for very controlled situations like translation, analyzing large amounts of text for key identifiers, seeing patterns between data sets.

I’m excited for the day when AI can enter our environment even more. We’re getting there.

I love the idea of combining software development with business and product development in business to business industries. We see many companies that need software development support but also need help with business development. Every company needs help with sales.

Often the software and business development are closely connected. Business development helps define product development which defines software development. We have a lot of business development experience. And I like the firms that offer growth services. Growth is such an important and educational process for every company. Having an intimate partner like a software development firm as part of the growth process could really help create new products.

For many years, product development firms have largely helped with product definition, customer research and defining the experience. This is all upfront, very expensive work. What happens when the product starts to sell in the market? There is no feedback loop.

Pretend we’re helping a company that sells software to car dealerships. Upfront we could talk to the car dealerships to understand their needs, use cases. This will help define the product. Then we build it, test it and deploy it.

This is where the fun really begins – asking what the customer thinks of it. Rarely will the first iteration stand up perfectly against a client’s wishes. That’s OK. Think of your first generation product, or a major update, as a trial run.

Build it in a smart way by gathering qualitative and quantitative measures from the customer. Then build it. Then let’s see what the customer thinks.

This is classic lean startup methodology, nothing new here. What is a little different is including Augment in the sales process. A true partner model where we help build the software and work to sell it with you to clients.

We have a very good platform for reaching out to potential clients. Maybe we should test it out with our b to b clients. Could be fun, interesting, educational and rewarding for all.

So you have an idea for a software product. Now what? In this post, I focus on the initial research around the technology and idea. In the next blog post I’ll talk more about the user experience, marketing, design/development and working with strategic partners.

I often run into researchers who have a great idea and technology, but are not sure how to commercialize their idea. Let’s focus on software. Although, many of these projects also include hardware too.

There is no easy answer on how to commercialize. In general, it’s a lot harder, takes a lot longer and costs a lot more money than expected. That’s almost a given. And that’s the downside.
The upside is that creating and launching a new product is inspiring, fun and makes you feel alive. Like all the bloggers out there say, we were meant to create.

Let’s take a look at an example. Let’s say a researcher has trained a machine to understand exactly what food you’re eating based on a picture from a cell phone. This would be pretty cool. There are some apps out there that are trying to do this.

What would I do to understand the potential for this technology? These steps aren’t always in this exact order.

Search Google Play:

Search the google play store to see what apps are already out there. Here are a couple:

Neither of these looks like they’ve solved the image recognition problem.

Google Search for Competing Technologies:

Next I’d try different google search terms to try to find competing technology, for example: “phone app that recognizes food”.

Here are a couple of promising apps: SRI’s Food Recognition Technology and CamFind.

These apps look like direct competitors to our example technology, but they are not necessarily in commercial production. I would definitely download and try out the CamFind. It claims it can recognize anything. Recognizing is one thing but understanding portions of food on a plate is another.

Search Google Patents:

Then, I’d do a quick patent search on google patents. If I search for “recognize food portions” a whole mess of results comes up. It takes a lot of weeding through the results to find relevant patents. Once you find patents relevant to your idea, you can read the claims and see who filed the patent. That can lead you to other companies working on the same idea. Maybe a potential partner?

Searching patents can be intimidating. It will look like your exact idea is taken. As you dive deeper, you may realize there are nuances where your idea might be different.

When I searched for our example idea on Google Patents, not a lot of relevant results came up. Keep in mind this is just one search. It’s best to do multiple searches, refining the keywords, sifting through the results.

Ask the Experts:

Contact a few nutritionists to understand what is state-of-the-art for recognizing food portions. And do they think an idea like yours would be helpful. I would contact maybe 10 people, if possible. With follow-up emails, I would guess that three people would respond to you.

I’d compose the email like this:

Hi Jane,

I’m emailing you because you’re a nutritionist at Awesome Nutrition company. I’m curious about your opinion on a new technology we developed that measures food portions via a simple picture.

I know you’ve been a nutritionist for about 10 years. I’d love your opinion on our technology, and to better understand the food portion landscape better.

I’m also wondering if this idea would even be useful.

Thanks for your time.

Brainstorm:

Brainstorm names for the application, and ideas for how it would work. This helps to make the technology come alive. It will help you understand better what is needed from the marketing and user experience perspective.

Maybe we call our idea Portion Labs. Then check if portionlabs.io is available. Then I would think about what features would make this app super useful. And what features could also help market it. Why would a user send something to a friend? Of course it should have social tie ins. But what else? It would be nice to connect to a database to understand all the nutrients you’re getting, and what the recommended daily allowances are. Are you getting enough nutrients?

It could also be helpful to have a buddy option where you hold each other accountable for life. This isn’t a diet app, it’s a life changing food app. Over time users will understand portions better and may not need it.

In the next blog post I’ll talk more about how to think deeper about the user experience, marketing and finding strategic partners.

A product team for hire. That’s what a lot of consulting firms offer. This includes the project manager, quality analysts, software developers, software architect/analyst, designer and potentially the product manager. Often the client will provide the product manager.

We’re working with some clients on this exact structure. It’s great. And the beauty of offshoring, besides saving money, is that you can scale faster. Talent is hard to find in the United States. It’s also hard in India but it’s a little easier.

So how does a product team work? It’s similar to how product teams work across Silicon Valley and both large and small companies. Let’s break down each of the roles.

Product Manager:

This is like the CEO of the product or project. They make the final decisions about design or functionality. Let’s use gmail as a product example. At google, I’m guessing they have a product manager in charge of gmail. They are responsible for the entire product. This includes the product development team and also sales/marketing.

The product manager manages the entire team. They lead the meetings and are ultimately responsible for the performance of gmail.

Project manager:

The project manager ensures things get done on time. They are also responsible for looking into the future and see what needs to be done now, what potential road blocks are there, what questions need to be answered now so that work isn’t slowed down.

They often have some technical know-how but that’s not always the case.

Quality Analysts:

QAs are responsible for testing the software. They’re a part of the regular meetings from day 1 because the QA will write and execute the test plans. If there is a bug in the program, the QA needs to find it.

Once the QA finds a bug, it’s added back as a task to be fixed in the next sprint, or sooner.

Software Developers:

And of course the software developers code the requirements. Often there are multiple developers on a product team. They’re assigned tasks by the project manager.

It’s important that the software developers understand the vision of the product and what each task means to the overall project.

Software architect/analyst:

The architect helps to define the requirements: what does the client want the product to do and technically how do we make that happen. For gmail, maybe we want to add a section for a calendar. If that’s the case, where will the calendar go (designer will help with this) and what functionality is needed?

Once the requirements are written for the calendar within gmail, then the developers code it and the QAs test it. Once the testing is done on the UAT server, the code is moved to the production server.

For a larger project, the architect will also write some overall project requirements around the code. The architect will design the database (sometimes a database designer does this) and establish security protocols and structure to ensure the product will be secure.

Designer:

The designer along with the product manager helps to set the vision for how the product will look. The designer will also be in charge of user experience (UX). A good designer helps set the tone for a project. They’re essential. Bad design will ruin products. It doesn’t matter how well a well a product is coded. If the design is off, it won’t work.

So that’s a brief overview of the product team. These teams are what create much of what we use every day: gmail, facebook (many product teams there of course), Amazon echo.

This is another virtual reality post. As you can tell, I’m definitely interested in its potential. And it’s more fun than writing about data integration or writing a new mobile app. Both of those are vital too. And it’ll be interesting to see what happens with data integration between third parties and the blockchain. That’s another post.

This post is on instilling a sense of fun into virtual reality applications. I’m almost done with the book Ready Player One. It’s brilliant. It’s set in the near future around a virtual reality world called OASIS. I won’t go into the entire plot, but if you have any interest in virtual reality, it’s a fun and intriguing read.

That’s the thing with virtual reality. It could potentially be used to solve real world problems, but there will always be that sense of gaming, fun involved with it.

For example, let’s say a company manufacturers large sewer pipes. It’s tough to bring examples of large sewer pipes to a sales meeting. The sales reps could show pictures. Boring, and not really the same as seeing them in person. With virtual reality, they could create an application so that the client can see the sewer pipe from all angles. The sales rep could walk them through multiple installations. How does it fit? How is it installed? How is it manufactured (with virtual reality they could also easily show the manufacturing process)?

Now instead of a powerpoint, you get a fully immersive presentation in virtual reality. The only thing they can’t do is touch it. But they actually could if the sales rep brought in samples.
The virtual reality tour could also include how an installation would fit within the client’s project. They could see exactly how it’ll fit, the timeline and any potential issues.

So what about the fun? There is so much you could do during the virtual reality presentation to include fun. You could have the sewer pipes flood with water when you’re walking through them. Or you could have a little rat follow you along for the tour. Or maybe a Teenage Mutant Ninja alligator (copyright issues with Turtles) leads the tour.

Or instead of having cranes lower the sewer pipe in, you could have a WWE wrestler. Or instead of using a welding torch, you could use someone’s eyes to melt metal (like a certain superhero perhaps).

The tour needs to be a serious and accurate demonstration of the sewer pipe. But we’re all kids inside. Think about a demo at a trade show. Which sewer pipe demo will draw a larger crowd. One that is very straightforward with the typical powerpoint presentation or one that includes an alligator-led virtual reality tour?

I’ve been watching a lot of videos on VR design. In this post I’ll talk about a video by Mike Alger.

Mike made this video as part of this Master’s thesis, I think. He’s now a VR designer at Google, not a huge surprise there. The video is quite good.

VR design is really interesting. I’m used to think about design on a screen. I’m not a professional designer, luckily we have a great team for that, but I understand how it works on a screen. How to get a user’s attention, make the user experience easy and fun, and how to direct a user’s attention to certain areas.

VR design in some ways changes that completely.

We’re used to seeing websites like this:

We have one (or maybe more) window on the screen right in front of us. If we want to see multiple windows at full or at least workable size, we need multiple monitors.

Now here’s an example of what our computer experience might look like using VR.

In this screen shot, the windows look curved, but in a VR headset they are merely surrounding us.

Now all of a sudden we have a huge amount of space where we can design our webpages. There is essentially 360 degrees of designable space. VR design could have a huge impact on architecture and interior design. Maybe that’s why a large number of early VR designers have an architecture background.

There are many new design aspects to consider with VR compared to traditional UI design. Here are some of them. This is not an exhaustive list may any means:

1. Angles: A person’s visual field only contains so many degrees. I’m not very good at looking behind me without turning around. And peripheral vision is good for picking up movement but not good for concentrating. VR designers can use the space behind and on the side but the focus will be in front.

Here’s a screenshot taken from Mike’s video:

For front design, what elements should be at which locations/degrees? Straight ahead is typically where the user looks. That should be where the most essential elements exist. But what about elements not directly straight ahead? Should they distract the user so the user looks at it? How much distraction is appropriate?

There are so many fascinating questions. One Hollywood director is very concerned about VR because for the first time we’re not sure where the audience will be looking. What if a major event happens in a movie and the audience is looking behind them and misses the major event. Oops. Can we rewind this thing?

VR designers will need to incorporate cues, saying “Look at me, look at me”, with a bit more subtlety I hope.

2. Mobility: When I’m talking about VR mobility I’m not talking a mobile phone. Instead I’m talking about the user walking around. So now the designer needs to think about angles but also where the user might go in the application. The options are infinite, making VR awesome and intimidating. VR designers will have to become immersive story tellers.

Here is another screen shot from Mike’s presentation:

The possibilities for interaction are endless. For example, let’s pretend we’re building a cooking show using VR video. Currently we watch as if we are in the room sitting across from the chef. We are stationary. But what if this was shot from the perspective of an actual chef in training?

Maybe it starts with looking through a recipe. The user could walk around the “kitchen” to look in the oven or at what’s cooking on the stove. How do you make sure the viewer is watching what they’re supposed to at the right time? Verbal and visual cues as to where to look will be essential.

What about the mobility aspect? To the side could be a list of ingredients needed and maybe the viewer could go and see how to buy the proper food. In this segment, the user could tag along and see the chef going to the farmer’s market. And then we could jump back into the kitchen.

This is similar to how you’d navigate a website, jumping around to different videos and sections. With VR the designer now has to determine the limitations. Where does exploration end? What limitations should be set?

What about designing an environment from the ground up? Now the limitations are endless. For the above example that would mean designing the kitchen and the farmer’s market using graphics, essentially creating a new virtual world. In the kitchen every little nook could be explored by the viewer. How do you control what you want the user to see? You don’t really, but you provide cues when something is happening that they should watch.

The VR designer will have to understand that what’s in the periphery is important and changes as the viewers moves their head around. It’s thrilling and intimidating to think about the possibilities.

The more we understand and experience virtual reality/augmented reality the more we think it will change industries. It has the potential to change retail and entertainment. Can you imagine watching a Packers game from any spot in the stadium? Amazing. Watching a football game on a pane of glass will feel like so yesterday.

It feels obvious that retail and entertainment will be disrupted. What about other industries? I would like to use some of these posts to process my thoughts around less obvious industries. This post is on analytics. I’ll write about both virtual reality and augmented reality.

Will analytics change with virtual reality and augmented reality?

I think so. But it’s not real obvious how. Let’s start with virtual reality. Right now analytics are often consumed via a dashboard. For example, if you’re looking at web traffic analytics, the dashboard could have hundreds of analytics including the number of unique visitors, country where they’re from, time spent on the site, where they came from, and many more. It’s often overwhelming.

Would a google analytics in VR make sense? How would it look? I think the big value for analytics in VR will be layers. In the above analytics I mentioned, country of origin could be one layer, time spent another, source of traffic another, number of visitors another. It would be like peeling a layer back from an onion.

How does this help provide deeper insights?

With this layered approach, I think you’ll be able to see new patterns. You could get in the middle of your data, literally. You could develop the VR app to help see these patterns and set visible alerts around trending valuable patterns. I think there is so much you could do but this would be a simple start.

What about augmented reality and analytics?

In the long run, augmented reality will replace much of what virtual reality does now. But what scenarios are possible right now?

What could be helpful are visual analytics. Imagine walking down a grocery aisle, looking at products, and seeing the sales of each product as you walk down the aisle. This would be a pretty simple analytics. Not sure how helpful this would be.

Another example is in a warehouse and visually seeing how many items are left in stock as you walk down the aisle.

Another example is taking analytics, lets’ say google analytics dashboards, and making them virtual using augmented reality. That way maybe you can see 5 screens at a time. This could be helpful for detecting trends. But still not as interesting as virtual reality.

I think that’s enough for now. The augmented reality examples bore me, so must be even more boring to you.

I always thought virtual reality was overhyped. Then I had a cutting edge demo around spaces like an office and oil rig. It blew my mind. That was two weeks ago. I still think about it all the time.

I now have so many ideas how virtual reality, then augmented reality, will change everything.

I could write 15 blog posts on the topic. In this post I’ll just focus on one over-arching VR application: social presence.

I have a lot of conference calls every week. As we all know, it’s always a mess trying to get everyone on the phone. And it’s quite impersonal. Face to face is still the best but it’s becoming less important. VR will make it even less important. I can’t wait until we all join a conference call with our VR headset. We can look into each other’s eyes and almost feel another person’s presence. We still have a long way to go though.

Probably the hardest part is looking someone in the eyes. Remember they have a large VR/AR headset. How are you supposed to see their eyes? Luckily researchers and companies are working on it.
For our podcast, I interviewed Hao Li, a brilliant graphics/AI researcher at Berkeley. He has developed some algorithms using an IR camera to track people’s face movements around the eyes. Right now it works well on an avatar. That’s the first stage of this technology, using an avatar.

The next stage will be superimposing the person’s actual face over VR/AR head set. That technology is not there yet but it’s coming. It will transform how we do business, meet people, interact with each other.

Beyond meetings it could work in so many different areas like concerts, games, conferences, events, anywhere where you want to meet and talk to other people. We’ll be able to just sit in our room at home and hang out with people across the world. Sounds nice. Also sounds a little scary. Will we all just become physical hermits, and virtual social butterflies?

Could you take a trip to visit a city virtually? Sounds lame but maybe not if it’s a specially designed virtual city. Think Second Life taken to another level. Would it be as enjoyable as in real life? Virtual ice cream doesn’t sound as tasty.

A lot of questions to answer.