Creating an Augmented Reality App using Papervision3D and FLARtoolkit. Part 1: Setting Up

Jersey Boys 3D

This project began when we found out that our client was going to be printing some paper fans to give away at Jersey Boys shows throughout the country. They were looking for something interesting to put on the fans, a link to a new site that fans would visit. We came up with the idea of printing a marker on the fans that could then be used at a specific website in order to produce a 3d, augmented reality image via the user’s webcam. We got the go-ahead on the project and the fans were sent for printing. The timeline was set and work began.
Click here to view the final project online and try it out for yourself. Read on to learn about our design process and setting up the project in AS3.

Before doing any coding or asset creation on the project, we started with storyboards in order to come up with ideas of what we wanted to happen when the viewer held up the fan to their webcams. We ended up doing two versions and then combining ideas from both into one. Here is the final storyboard we used, along with description text and arrows to indicate movement (yes, we went old school and hand-drew the graphics):

 

Storyboard for Jersey Boys 3d project

After walking the client through our storyboard, we got the ok to build it out. Now the real work began. We started by looking into the technologies that we would be using in the project. We had done some work with Papervision (http://blog.papervision3d.org/) in the past so we were familiar with how it worked. We made sure we had the latest version of Papervision 3d (via SVN here) and set up our file structure for the project. Next we downloaded the FLARtoolkit files, available here, the technology we would use to attach our 3d Flash movie onto the marker on the fan in 3d space. After downloading and placing these files into our project, we got to work.

We start by creating an Actionscript file that would handle the attachment of the 3d object to the marker. So fire up Flash or Flex and start a new Actionscript file (I am naming mine “MainFlar.as”). First things first, the imports.
To start, import all of the flash classes we will need (be sure and change the paths to match the class structure of your project):

[as3]
import flash.display.BitmapData;
import flash.display.Sprite;
import flash.events.Event;
import flash.media.Camera;
import flash.media.Video;
import flash.utils.ByteArray;
import flash.display.StageQuality;
import flash.filters.ColorMatrixFilter;
import flash.utils.Timer;
import flash.events.TimerEvent;

//Next, all of the Papervision 3D classses
import org.papervision3d.lights.PointLight3D;
import org.papervision3d.materials.shadematerials.FlatShadeMaterial;
import org.papervision3d.materials.utils.MaterialsList;
import org.papervision3d.objects.primitives.Cube;
import org.papervision3d.render.BasicRenderEngine;
import org.papervision3d.render.LazyRenderEngine;
import org.papervision3d.scenes.Scene3D;
import org.papervision3d.view.Viewport3D;
import org.papervision3d.objects.DisplayObject3D;
import org.papervision3d.events.RendererEvent;

//And finally, all of the FLARtoolkit classes:
import org.libspark.flartoolkit.core.FLARCode;
import org.libspark.flartoolkit.core.param.FLARParam;
import org.libspark.flartoolkit.core.raster.rgb.FLARRgbRaster_BitmapData;
import org.libspark.flartoolkit.core.transmat.FLARTransMatResult;
import org.libspark.flartoolkit.detector.FLARSingleMarkerDetector;
import org.libspark.flartoolkit.pv3d.FLARBaseNode;
import org.libspark.flartoolkit.pv3d.FLARCamera3D;

[/as3]

Ok, now we need to create 2 files in order for the code to have something to detect: 1) the pattern file. This is a .pat file that is basically a representation of the marker you are using. They are online services that will generate this file for you when you upload your marker file, here is one. There is also an air app to do this, available here. 2) The “camera_para.dat” file. This is a binary file that is used to calibrate your webcam. For this you should be fine just using the one that comes bundled with the fLARtooolkit.

Back in our MainFlar.as file, we now need to embed these 2 files for use in the project, and create classes for them (don’t forget to change the paths to match those of your project):

[as3]

[Embed(source="../assets/pattern/marker.pat",mimeType="application/octet-stream")]
private var _MarkerPattern:Class;
[Embed(source="../assets/camera_para.dat",mimeType="application/octet-stream")]
private var _CameraParameters:Class;
//Ok, now to set up your variables (these will be explained below):
//vars for the camera to capture the video stream
private var _myCamera:Camera;
private var _myVideo:Video;
private var _capture:BitmapData;
//FLAR vars that will detect and attach our marker
private var _cameraParams:FLARParam;
private var _markerPattern:FLARCode;
private var _raster:FLARRgbRaster_BitmapData;
private var _detector:FLARSingleMarkerDetector;
private var _cam3D:FLARCamera3D;

//PV3D vars
private var _scene3D:Scene3D;
private var _viewPort:Viewport3D;
private var _mainContainer:FLARBaseNode;
private var _renderer:LazyRenderEngine;
private var _trans:FLARTransMatResult;

//our PV3D Display Object that contains all of our 3d objects
private var _main:MainDisplay=new MainDisplay();
[/as3]

Ok, now that that is set up, we need to create a Papervison 3D object that we can use to attach to the marker.
I used a MainDisplay class (see above) that extends the DisplayObject3D PV3D class. That way I could add various elements to my display and have one central class to keep everything in. I separated each element I would need into a separate class that I would then add to the MainDisplay class. For example, I created  a Flashbulb class that creates a spotlight at a specific size. I added a public function that would then return this object to my MainDisplay class. Here is the code for the Flashbulb class:

[as3]
public class Flashbulb{
private var _flashbulb:Sphere;
private var _colorMaterial:ColorMaterial;
private var _glow:GlowFilter;

//Initialization
public function Flashbulb() {
super();
_colorMaterial=new ColorMaterial(0xFFFF00);
_glow = new GlowFilter(0xFFFFFF, 1, 10, 10, 4, 3);
_flashbulb=new Sphere(_colorMaterial,10, 10,20);
_flashbulb.useOwnContainer = true;
_flashbulb.filters = [_glow ];

}
//function to return this object to my MainDisplay class
public function getBulb() {
return _flashbulb;
}

}
[/as3]

Back in MainDisplay.as, I declared the variable for the new Flashbulb object and called the function that would return this object:

[as3]
private var _Flash1:Flashbulb = new Flashbulb();
private var _introFlash1:Sphere=_Flash1.getBulb();
[/as3]
Then I added the _introFlash1 object to my MainDisplay.as file, placed them in their proper position and set their initial properties (which would be animated later):
[as3]
this.addChild(_introFlash1);
_introFlash1.x=100;
_introFlash1.z=200;
_introFlash1.y=100;
_introFlash1.scale=.01;
_introFlash1.alpha=0;
[/as3]

Another advantage of using this method was that it allowed me to create multiple instances of the Flashbulb object for use in MainDisplay.Back in FlarSpace.as, I was ready to setup my camera, initialize the FLAR settings and add the MainDisplay to my marker.First I called a findCam function that looked for the user’s USB class video camera (the preferred camera for accessing the viewer’s webcam). If it wasn’t found, it would get whatever camera the user has installed and attempt to use that.

[as3]
private function findCam():void{
for (var i:int = 0;i<Camera.names.length;i++) {
if (Camera.names[ i ] == "USB Video Class Video") {
_camIndex = i;
_myCamera=Camera.getCamera(String(_camIndex));
} else {
_myCamera=Camera.getCamera();
}
if (i== (Camera.names.length-1)){
setupCamera();
}
}
}
[/as3]

Once a camera was found, I called the setupCamera function. This function sets up the camera and attaches it to my video object:
[as3]
private function setupCamera():void {
//these dimensions shoudl match those of your main swf
_myVideo=new Video(640,480);
//look for camera object, if one is not found, display warning
if (_myCamera==null) {
addChild(_warning);
trace("no camera");
}
//this sets the resolution and framerate of the camera
_myCamera.setMode(320, 240, 15);
//attach the camera to your Video object and add it to the stage
_myVideo.attachCamera(_myCamera);
this.addChild(_myVideo);
}
[/as3]
Next, I initialized the FLAR settings in the following functions.
The first function sets it up to use the marker pattern that I had embedded earlier:
[as3]
private function setupFLAR():void {
_cameraParams = new FLARParam();
_cameraParams.loadARParam(new _CameraParameters() as ByteArray);
_markerPattern=new FLARCode(16,16);
_markerPattern.loadARPatt(new _MarkerPattern());
_trans = new FLARTransMatResult();
}
[/as3]
The next function sets up the bitmap data that would allow the camera to detect the marker:

[as3]
private function setupBitmap():void {
_capture=new BitmapData(640,480);
_capture.draw(_myVideo);
_raster=new FLARRgbRaster_BitmapData(_capture);
_detector=new FLARSingleMarkerDetector(_cameraParams,_markerPattern,80);
}
[/as3]
Once I had my camera and marker all set up, it was time to set up my PV3D scene and bring in my PV3D Display Object (the “_main” var  above that was typed to the MainDisplay class I created earlier):

[as3]
private function setupPV3D():void {
//initialize PV3D vars
_scene3D = new Scene3D();
_cam3D=new FLARCamera3D(_cameraParams);
//initialize the FLARBaseNode container
_mainContainer = new FLARBaseNode();
//attach the FLARBaseNode that will attach to the marker to our PV3D scene
_scene3D.addChild(_mainContainer);
//initialize the PV3D Renderer
_renderer=new LazyRenderEngine(_scene3D,_cam3D,_viewPort);
//initialize the PV3D Viewport and add it to the stage
_viewPort = new Viewport3D(640,480);
this.addChild(_viewPort);
//this is where we attach our _main (MainDisplay object) to the FLARBaseNode
_mainContainer.addChild(_main);
}
[/as3]

That’s it for Part 1. In Part 2, we will set up the listeners that listen for when the marker is detected (and undetected). We will also get into how the animation is triggered and stopped. Tune in soon for Part 2, thanks for reading.

About Curious Minds
We are a web development firm in New York and Chicago, providing development resources and consulting for websites and mobile apps since 2004.