Skip to content

Latest commit

 

History

History
153 lines (93 loc) · 7.79 KB

isaac_setup.md

File metadata and controls

153 lines (93 loc) · 7.79 KB

Set Up the NVIDIA Isaac Simulator

We used the Isaac Sim 1.2 for the photo-realistic simulation. We mainly used the provided environment and the Unreal Engine used by the simulator. Therefore, it should be easy to use other Unreal Engine-based simulators instead.

Note that ISAAC 2019.2 is used, as it is the version available when we set up the experiments. Since then, there seems to be significant changes to the simulation setup from NVIDIA (old UE4 simulator discontinued and the switch to the new simulator). Unfortunately, the simulator we used is not supported by NVIDIA anymore. Therefore there might be issues with new versions of drivers and CUDA.

Install

For the first two steps, we simply set up the Isaac Sim 1.2 and UnrealCV. And in the last step we replace the shipped warehouse environment with our modified one and rebuild the map for rendering. Make sure that your machine satisfies the requirement.

Step 0 Dependencies

The simulator is tested on Ubuntu 18.04 with the following setup:

  • NVIDIA driver 440.33 (this is the version bundled with CUDA 10.2): install with sudo apt install nvidia-driver-440

  • CUDA 10.2: install according to the instruction here

  • Vulkan 1.1.108.0: please download here and follow the instruction here to setup

Step 1 Install Isaac Sim 1.2

Below is a summary of the official instruction.

First, download the Isaac Sim description file from here (i.e., the Isaac 2019.2 release) and extract contained xml file.

Then run the following to setup the simulator

git clone --single-branch --branch IsaacSim_1.2 [email protected]:NvPhysX/UnrealEngine.git
cd UnrealEngine
rm Engine/Build/IsaacSimProject_1.2_Core.gitdeps.xml
cp <extracted_xml> ./Engine/Build
# this may take a while
./Setup.sh
./GenerateProjectFiles.sh
./GenerateTestRobotPaths.sh
# this will again take a while
make && make IsaacSimProjectEditor

Verify

Start the simulator

# this will take a while for the first time
./Engine/Binaries/Linux/UE4Editor IsaacSimProject CarterWarehouse_P

You should see a warehouse-like environment.

The official instruction uses Vulkan (-vulkan). This however makes the editor crash constantly due to VK_ERROR_DEVICE_LOST error in our test setup. This error seems to be related to driver versions, and we could not identify the exact cause. Without -vulkan, OpenGL will be used and is sufficient for our purpose. You can try with -vulkan on your setup.

Step 2 Install UnrealCV from source

Please following the official instruction to install from the source. Remember to replace the --UE4 argument with where you place the simulator.

Verify

Start the simulator as before

./Engine/Binaries/Linux/UE4Editor IsaacSimProject CarterWarehouse_P

Start a python interpreter

import unrealcv
unrealcv.client.connect()

and you should see a message about successful connection.

Step 3 The modified warehouse environment

Download the modified warehouse environment from here and replace the folder in IsaacSimProject/Content/Maps/Warehouse with the one in the downloaded zip file. Then start the editor as before, you should see a modified warehouse environment that looks like

warehouse_env

If you want to rebuild the environment, e.g., after making some changes, you should increase the number of allowed open file (before starting UE4Editor, in the same terminal):

ulimit -n 10000

Using the Simulator

As mentioned before, you can start the simulator from the command-line using

./Engine/Binaries/Linux/UE4Editor IsaacSimProject CarterWarehouse_P

We will mostly use the simulator as a server to provide rendering functionalities. For this you need to click the Play button (on top right of the view port) and then leave the editor running in the background. The interaction will be done via the UnrealCV plugin and the unrealcv_bridge package.

Changing Camera Intrinsics for UnrealCV

You need to change the configuration file before you start the simulator

Note that the visualization you see is using the settings from the editor, which is not the same as the rendering settings. Since we use the UnrealCV, the only reliable way of changing the camera parameters seems to be via the configuration file Engine/Binaries/Linux/unrealcv.ini, which looks like:

[UnrealCV.Core]                                                           
Port=9000                                                                 
Width=640                                                                 
Height=480                                                                
FOV=90.000000                                                             
EnableInput=True                                                          
EnableRightEye=False

Please read this thread for some more details.

Note that UnrealCV also provides command line interface to change the http://docs.unrealcv.org/en/latest/plugin/config.html and horizontal FoV. We could change the FoV but not the resolution via its command system. Therefore we resort to the configuration file.

Packaging

We can also package the map to be a standalone application, without relying on the Unreal Engine Editor.

First, add the following line to Engine\Config\ConsoleVariables.ini

r.ForceDebugViewModes = 1

for UnrealCV to work properly.

Second, in the Editor, change the project's default map to CarterWarehouse_P according to the instruction here.

Third, only select the map you want to package (here CarterWarehouse_P) under Project Settings -> Packaging (i.e., delete the other maps).

Finally, use File -> Package Project -> Linux to start package - and this is how we created the provided standalone simulator.

Please refer to the official documentation and UnrealCV documentation for more details.