Digital Twin
Overview
You can test your operations and robot’s work in Aitheon Sim software, which is a digital playground embedded in Smart Infrastructure.
This allows you to make a prototype and test your infrastructure with Aitheon Core and simulated hardware and robots even in it prior to building a facility and actually buying robots for it.
Here's the general flow for setting up the simulation:
Load a CAD designed robot model -> set up properties for its parts as for physical objects -> set up joints between immovable and movable parts -> set up sensors -> set up a scene for the simulation.
In the following tutorial, we are using the Yezhik robot model by Aitheon as an example. Please, be aware that all the properties` values here are for this particular model.
Aitheon robots simulation models are available in Marketplace, already configured for simulations.
Steps to Make a Digital Twin
Import a robot model
Create a collision scene
Set up physics scene
Prepare robot to reflect physics
Set up robot`s joint
Add sensors and measurements
Create and link camera
Debug
Add a robot to a scene
Give task and test
Avoid possible issues
1. Import a Robot Model
To load a CAD software designed robot model (STEP format):
1) Go to menu Window->Isaac->Step importer and pick a .stp file.
Depending on the model detailing it may take a while.
2) When a Step import window will appear, scroll it down and click Finish Import and choose a directory to save converted model objects.
3) Go to menu File->Save As and save the model as a single USD - this is the 3-d model format that the simulator uses. After that, you can easily open this model by loading this file: menu File->Open.
The right Stage tab contains all the objects in the tree. The imported robot is the Root object in this instance.
4) Change the view if needed: toggle Perspective to Top, Front, or Right view in Viewport. Drag the view with the right mouse button pressed and hit F after these moves to center the view back to the chosen object (part). The mouse wheel zooms the view in and out.
5) Change the position if needed: switch to Rotate selection mode in Viewport and drag the sphere or go to the Details tab and change rotation and position numbers.
6) Group minor parts into bigger containers. Choose parts holding Ctrl, right-click, and hit Group Selected. All immovable parts can be joined in the chassis group, for example. Also, you can drag-n-drop elements to a group.
7) Also, for your convenience right-click and rename parts and groups of the object.
2. Create a Collision Scene
We need real physics in our simulation to test the robot. So we have to add a physics scene that imitates it.
1) Go to menu Physics->Add->Physics Scene.
In the Stage tab, the World object will appear.
2) In the same way add ground: Physics->Add->Ground Plane.
It's a basic plane for the robot`s collision with the environment.
Note: ground plane appears at the Stage tab in the Root object (the robot model object name in our example) and is called staticPlaneActor. So when you manipulate the robot model, the ground plane will move with it. To unleash this plane and bind it to the World object - drag and drop staticPlaneActor from Root to World or the higher level (as Root and World).
Adjust Position
Choose an object (root or staticPlaneActor) in the Stage tab to adjust its position and in the Details tab specify needed coordinates.
3. Set Up Physics Scene
We need to set up the physics scene in order to receive feedback from the simulator environment.
Expand the robot model object in the Stage tab (in our example it is called Root), find and click physicsScene.
Choose PhysX Properties lower-right tab.
Remove enableGPUDynamics flag.
Set collisionSystem: PCM, solverType: PGS, broadphaseType: MBP.
Usually, default presets are ok, but these settings work better.
4. Prepare Robot to Reflect Physics
To reflect physics an object (robot model) should have RigidBodies properties. Apply these properties to every element (or group of elements) of the object, but not to the top wrapper (here called Root).
1) Click on each part (a group of parts) that will interfere with the environment and add the property: Physics->Set->Rigid Body. In the PhysX Properties tab new properties - Physic Body and PhysX Rigid Body - will appear.
Do it one by one with all the parts (or groups). It will not work for multi-selected parts (and groups) in most cases.
If you first give all the parts Rigid Body properties and then group them, these parts will fall apart during the simulation, because the system treats them as separate. So it's better to group them first and then give Rigid Bodies to the whole group. Another solution: you can add Fixed joints to these separate parts (see the Joints chapter).
2) In the PhysX Properties tab of each element that you want to participate in collisions go to Physics Prim Components and Add Prim Components: CollisionAPI and MassAPI:mass.
MassAPI:mass will allow you to specify the Mass Properties. Otherwise, the defaults will be used (the part`s geometry multiplied by a density of 1000).
In the chassis part of the robot, there are a few elements that interact with the environment. So you may want to delete the CollisionAPI from internal elements (this will make the simulation “lighter”):
3) Choose the chassis group and go to Physics->Remove->Collider. This will remove all the collision APIs.
4) Select and apply CollisionAPI in PhysX Properties to external elements one by one.
5. Set Up Robot`s Joints
Joints give you the ability to connect rigid bodies in ways that are not entirely rigid. A good example is a car: the wheels of the car revolve around their respective axes, the suspension can slide up and down along an axis. The technical term for the former is “revolute joint”, the second is a “prismatic joint”.
To add a joint to your scene, first select the two objects to connect. It is also possible to select a single joint in order to connect it to the world frame. Then select Physics->Create->Joint. In a submenu, you will be able to select the type of joint.
Articulations is an advanced, hierarchic mode of joints, useful for creating hierarchical mechanisms such as a vehicle or a robot. To use articulations, one should organize the joints of a mechanism and the objects they connect into a tree structure. For example, to create an articulated wheelbarrow, one would create the body (tray) object, which would have a child revolute joint for the wheel axis, and the joint would have a child wheel body. Articulated joint links parts starting from the articulation root to the last chained connection. The top tree wrapper should have ArticulationAPI in order to work correctly in the future.
The graph of joints connecting bodies will be parsed starting at this body, and the parts will be simulated relative to one another, which is more accurate than conventional jointed simulation.
There are several types of joints, mainly used are RevoluteJoint or just PhysicsJoint (it`s a basic type of joint without additional API`s ).
A description of joints and their behavior can be found here.
PhysicsJoint is not listed here because there is no such type in the menu. This type is basic and concerns every other type. Usually, we use this type for a root joint with Articulation Joint in PhysX Properties.
A description of Articulated Joints can be found here.
Step 1. Apply ArticulationAPI to top tree wrapper
Add a method of building the joints chain with the model.
1) Select top tree wrapper (Root).
2) In the PhysX Properties tab and add ArticulationAPI.
3) Set solverPositionIterationCount in PhysiX Articulation properties to 64.
4) Set solverVelocityIterationCount to 16.
You can set up these two parameters to higher numbers for better precision, but it will load the system a lot.
Step 2. Create ArticulatedRoot
Make a root object for all joints to connect to.
1) Select top tree wrapper (Root in our example).
2) Go to Physics ->Add ->Joint ->To World Space.
3) Select the newly created joint in the Stage tab.
4) Go to the PhysX Properties tab and Remove Join Component that is present there.
5) Add Joint Component named ArticulationJoint.
6) Scroll down to the Physics Articulation Joint property and change articulationType to articulatedRoot.
7) Add a tab to the editor: go to Window ->Isaac ->Relationship Editor.
8) A new Relationship Editor tab will appear, open body0, change the 0 path to your chassis object (for example, if you grouped all the chassis parts to chassis group in Root, the path will be /Root/chassis), and click Modify. Now the root object is attached to the chassis.
Step 3. Create Joints for movable parts
Create all joints for all movable parts of the robot (model). If there is an immovable part that wasn`t grouped with the rest immovable elements and got common Rigid Body properties you should create a joint for it too. Choose Fixed type in this case. If you don't do this the ungrouped and unjointed element will fall apart from the model during the simulation.
1) Select two Rigid Bodies: the primary one first, then the secondary one. A joint will be created as a part of the second component (and will appear in the second component`s submenu in the Stage tab).
2) Make sure you have the correct joint type set in Physics ->Joint Attributes->Type - Prismatic (or maybe Fixed).
3) Choose the connection type: Physics->Add->Joint->Between Selected.
4) Select the new joint in Stage and in the PhysX Properties tab Add Joint Component - ArticulationJoint API.
5) When you select the joint in Stage it gets visible on the Viewport tab. Move the joint to the correct place and apply the correct rotation (align its position and movement directions to actual elements of the model - use arrows dragging and Rotate mode sphere to move the joint). It doesn't have to be 100% precise, because the ArticulationJoint API will solve minor inaccuracies.
When you move the joint element with the mouse button pressed and then release it, the selection highlighting will jump to another object. To switch back to the joint selection press Ctrl+Z. Or click the joint element on Stage again.
6) In the joint`s PhysX Properties tab scroll to PhysX Joint and put a flag to enableCollision property.
7) If this is a joint for a driving wheel: add Drive API in PhysX Joint Components properties, scroll to Joint Drive properties, and set angular:targetType to velocity, angular:type to acceleration, and angular:damping to 10000.
Step 4. Repeat Step 3 for all the movable parts.
6. Add Sensors and Measurements
Add Lidar
Lidar is a special Isaac Sim component for measuring distances (though laser beam reflection measurements). Lidar beams in the simulation will ignore anything that doesn’t have a collision API attached to it.
1) In Stage (or in Viewport) select an object that represents the lidar, then click Create->Isaac->Sensors->Lidar.
To hide an element choose it and press H. To unhide - press again or go to Edit->Unhide all.
2) In the Details tab set the Z-axis Position on 1.3 to move the lidar up.
3) In the Other section of Details enable drawLidarLines and drawLidarPoints. It's optional, but useful for debugging. For example, if you start a simulation and don't see “laser beams” of the lidar, you didn't set up minRange properly (see next).
4) In Others set maxRange to 16, minRange to 0.08, rotationRate to 12.
These parameters are for the example model of the Yezhic robot. Use your values for your devices.
Add IMU
An inertial measurement unit (IMU) is an electronic device that measures and reports a body's specific force, angular rate, and sometimes the orientation of the body.
In Isaac Sim, IMU can be represented as a simple cube shape with rigid body properties, but with disabled collision.
If you already have IMU in your model, you can apply Rigid Body property to it and go to step 5 at once.
1) Create a shape inside the root wrapper: right-click on Root (or another name you gave to the root object), then Create->Shapes->Cube.
2) Select this Cube object and move it in the middle of the robot model in Viewport.
3) Scale it to the usual IMU size in the Details tab by changing Scale numbers on the X, Y, and Z axes.
You can slide the numbers in the axes fields - left and right by holding the left mouse button.
4) Make it invisible: Details tab->Other->purpose: guide.
5) Create a joint between the IMU and the chassis: select the IMU and the chassis group, Physics->Add->Joint (choose any type of it), in the PhysX Properties tab Remove Joint Component a present API and Add Joint Component - ArticulationJoint.
Creating REB Components
The Robot Engine Bridge (REB) extension enables message communication between the two platforms (Isaac Sim with Isaac SDK™) over TCP to perform a robot simulation. These messages include simulated sensor data, drive commands, ground truth state of simulated assets, and scenario management.
Mainly used REB components are:
Differential Base - for wheels moves simulation.
Lidar - lidar simulation.
RigidBodies Sync - objects interaction for multi-robots simulation.
Differential Base REB
1) Select the root object and go to Create->Isaac->Robot Engine->Differential Base. This will create REB_DifferentialBase in the Root object.
2) In the Relationship Editor tab set chassisPrim path to the top wrapper (/Root in our instance).
3) In Details set leftWheelJointName and rightWheelJointName accordingly (important: we type the wheels joints' names, but not wheels` names!) and press Enter after each to save.
4) In Details set the robot`s direction vector robotFront to 1, -2, -1.
5) If there is a Proportional gain field, it should be set to 3.
6) Set WheelBase to 0.406, WheelRadius to 0.1.
These parameters are for the example model of the Yezhic robot. Use your values for your devices. Also, you may want to change some default values, maxSpeed, for example.
Lidar REB
In Relationship Editor set the path for lidarPrim to the lidar component (created through Create->Isaac->Sensor->Lidar). Important: the lidar component in Stage must be Lidar type, not Mesh or else.
RigidBodies Sync REB
In Relationship Editor set path in rigidBodyPrims 0 to chassis, in rigidBodyPrims 1 to IMU.
7. Create and Link Camera
You can add extra points of view to your simulation by adding virtual cameras. For example, you can add a camera to the robot model and switch to its view.
1) Choose the view for a new camera and create the camera through Create->Camera in the Perspective menu of Viewpoint.
2) You can switch to the created Camera by picking it in the Perspective menu.
3) If you want to change the camera`s perspective, choose it, and move the point of view to the position from which you want to observe the real-time scene (holding the right mouse button).
4) To bind the camera to the robot (so that the point of view will follow the robot moves): select the camera object in the Stage tab, right-click and create a new group (consist of one element - the camera). This is done because the Rigid Body property can be applied only to Xform elements (see in Stage) - groups.
Apply Rigid Body properties to the created group (Physics->Set->Rigid Body).
5) Create a physics joint like for IMU.
8. Debug
In order to see collision shapes and debug this in real-time:
Go to Physics->PhysX Debug Window.
Move the tab to a convenient place for you.
In Show collision shapes pick Selected. If you choose All - the representation will become too “heavy”.
Collision shape movement is being shown only when you press the Step button, even if you run a scene from this window it won’t be constantly showing you collision form change.
9. Add a Robot to a Scene
Presumably, you want to test your virtual robot model in some virtual environment.
After you set up your robot virtual model and saved it in a .usd, create the scene, and add the robot to it:
Go to the Content tab, choose the model file, drag and drop it to the scene.
To connect the simulation to Smart Infrastructure go to the Robot Engine Bridge (1) tab and press Create Application (2). Then click 'play' (3) to start the scene:
10. Give a Task and Test
Add the robot virtual model as a device, and a map of the virtual scene as a floor to the Smart infrastructure service (See Add Floor and Add Device).
Launch an application to process the connection of the scene to Smart Infrastructure.
Create Task and watch the execution.
11. Possible Issues and Solutions
1) If the root joint will have a relationship to the wrong body (which doesn’t contain anything to link) it will crash i.e. if you forget to change the relationship after creating this joint.
What to do: make sure you change the relationship to be connected with the chassis.
2) Sometimes when you create/delete an element the system doesn’t correctly handle this and crashes when you start the scene.
What to do: save and reopen the scene.
3) The system will crash when creating a joint between bodies that are not rigid.
What to do: make sure that all bodies that you’re creating joints with are rigid (have a Rigid Body property).
Last updated
