Why Do Humble and Rolling Tutorials Require Building MoveIt From Source?

[davetcoleman]

In the moveit2_tutorials.repos it lists moveit.git as a dependency. I can imagine at one point something was out of sync from the debians so this was required. However should this be standard? It means our users must spend a lot of time building.

I don't feel strongly either way but the getting started tutorial does not clarify this is happening. It instead says on the rosdep step:

The following will install from Debian any package dependencies not already in your workspace. This is the step that will install MoveIt and all of its dependencies:

I am working on cleaning up that tutorial to clarify it is being built from source. But am not sure this is the best idea. @tylerjw

[davetcoleman]

I've been trying to build MoveIt, following our getting started tutorial, for over 30 minutes now. It's not been a fun experience.

[mikeferguson]

It took me 56 minutes to build JUST moveit2 and srdfdom (which had a change that I needed to use). It looks like latest debs are not past the test server yet for humble (so, needs to sync still before anyone can easily use debians)

[vatanaksoytezer]

We have released, but we are waiting a sync on Rolling and Humble. Stable branches (humble, galactic, foxy) of tutorials should use debians, main branch should use source since main branch of the tutorials can be updated with a change in main branch of moveit.

[vatanaksoytezer]

I've been trying to build MoveIt, following our getting started tutorial, for over 30 minutes now. It's not been a fun experience.

We actually have prebuilt docker images, that can people use that is documented here. In my opinion we should promote it.

[davetcoleman]

My colcon build keeps failing silently... I think I'm running out of memory because I'm in a VMware box with 8 GB ram (on a 32 GB laptop). It fails silently because my terminal just closes when it does. I'm not having a great "Getting Started" experience so far... still haven't gotten past this first tutorial.

I've been improving that tutorial here:

This build command will likely take a long time (20+ minutes) depending on your computer speed and amount of RAM available (we recommend 32 GB). If you are short on computer memory or generally your build is struggling to complete on your computer, you can append the argument --parallel-workers 1 to the colcon command above.

[davetcoleman]

We actually have prebuilt docker images, that can people use that is documented here. In my opinion we should promote it.

I can add this to the Getting Started tutorial now

[smartin015]

Hey folks - I love that there's now a Docker approach to getting started with MoveIt, but there's still a couple snags that will trip up users unfamiliar with Docker / Moveit / Ros2 etc. Here's my notes:

# After fetching the start-docker script and chmod +x
./start-docker.sh moveit2-humble moveit/moveit2:humble-source

Comes up just fine - thanks for this script, although I do wish there was a less scripty way of getting into a container (e.g. docker-compose.yaml). I'd gladly take a crack at contributing one if desired.

At this point I returned to the tutorial. I noticed moveit2_tutorials is not present in src/, so attempted to download and build it:

cd ~/ws_moveit/src && git clone https://github.com/ros-planning/moveit2_tutorials -b main
vcs import < moveit2_tutorials/moveit2_tutorials.repos

VCS asks me for my github credentials. Why? I'm just pulling public packages, aren't I? Also note that the workspace in the container is ~/ws_moveit, not ~/ws_moveit2` - this will likely trip some folks up.

Regardless, authentication fails with:

...
=== ./moveit2 (git) ===
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/ros-planning/moveit2/'

But the rest succeeds. Pushing bravely on:

rosdep install -r --from-paths . --ignore-src --rosdistro $ROS_DISTRO -y

Causes apt errors Unable to locate package. Running apt update and retrying works just fine.

cd .. && colcon build --mixin release

Build succeeded!

source ~/ws_moveit/install/setup.bash
ros2 pkg list | grep moveit2

shows moveit2_tutorials 🎉

Happy to contribute a PR for some of the simpler stuff (e.g. missing apt update), but I have no idea what's going on with VCS nor why everything is fine despite authentication failure.

(Edit: moving into the "First C++ MoveIt Project" tutorial gives me the feeling of missing a few legos from the kit. How do most folks use these containers when developing - mount a local directory with the package source? Maybe the start-docker script should also launch the container as the local user to prevent root permissions issues? I have hastly-formed opinions, but I'd prefer input from someone who uses this regularly)

(Edit 2: --mixin release takes a couple seconds, but --mixin debug has been running for 7 minutes so far and is still about half done)

[vatanaksoytezer]

Hello @smartin015 thank you so much for going through it and we really appreciate writing your thoughts and experience here.

Comes up just fine - thanks for this script, although I do wish there was a less scripty way of getting into a container (e.g. docker-compose.yaml). I'd gladly take a crack at contributing one if desired.

I would be more than happy to review and merge if you open up a pr using docker compose. The script was something I wrote a few years ago and hasn't been much updated since. docker compose has been on my mind for a long time, but never sat down to actually implement it for moveit.

At this point I returned to the tutorial. I noticed moveit2_tutorials is not present in src/, so attempted to download and build it:

That is expected since the docker images are meant for only moveit and moveit2_tutorials. We sometimes use this image as a base image for our development workspaces, which do not necessarily have tutorials. Having said that I'm more than happy to create a docker job in this repository that uses the distro-source image as a base image and adds and builds moveit2_tutorials to it.

VCS asks me for my github credentials. Why? I'm just pulling public packages, aren't I?

I am not very certain why this is the case but I can confirm. It's not about vcs afaik though, if you have tried to git pull you would have been asked the same. You can put a step that adds random credentials actually (as in https://github.com/vatanaksoytezer/ign-ci/blob/main/Dockerfile#L13-L14) which we can include in the new docker image for tutorials.

Also note that the workspace in the container is ~/ws_moveit, not ~/ws_moveit2` - this will likely trip some folks up.

Good catch! We are trying to use more moveit and less moveit2, we should perhaps change the documentation to use ws_moveit instead of ws_moveit2

Causes apt errors Unable to locate package.

That is very expected, since we remove apt lists after building a docker image to minimize the disk space it takes. This is a very common practice in docker images to minimize people's download time and keep the docker image as minimal as possible. We should document adding apt update.

Happy to contribute a PR for some of the simpler stuff (e.g. missing apt update), but I have no idea what's going on with VCS nor why everything is fine despite authentication failure.

We'd be delighted to have your contribution here. Just tag me in your PR and I'll help it get merged.

How do most folks use these containers when developing - mount a local directory with the package source?

It changes from people to people, I'll speak for myself. I do use dockers for most of my development from client projects to maintaining moveit across 6 different ROS distributions and 4 different Linux distros. What I usually do to create a dev workspace is I just create a new container from distro-source images, add my ssh and git configs, attach VS Code to my container to develop, create, build, develop everything in the container, then pull and push from the container. I feel that's much easier since docker is isolated from my local environment completely and my container is not going anywhere unless I delete it. I kinda like to use it as a fast, terminal version of a VM environment. But that's my liking I've seen other people mount workspaces and use local users as well. If you are going to mount directories root container is a bad idea I think.

Maybe the start-docker script should also launch the container as the local user to prevent root permissions issues?

Can you give a few examples on what root permission issues did you face? I'm happy to consider alternatives. I am assuming most people going over tutorials are not docker expert, which I think root gives much less learning curve using docker.

(Edit 2: --mixin release takes a couple seconds, but --mixin debug has been running for 7 minutes so far and is still about half done)

I'm assuming release had been cached already and debug is not. Having said that I'm not very surprised with that debug build take much longer time than release builds since they contain large debug symbols. For context moveit (1) CI uses debug builds for ccov and release builds for the rest which takes 2 hours versus 50~ minutes (see https://github.com/ros-planning/moveit/runs/6660340188?check_suite_focus=true).

@smartin015 Please let me know if this clarifies things for you. Thank you very much for the write-up and we would be happy to have your contributions here! I'll try to fix one issue at a time from this post whenever I can as well.

[smartin015]

I would be more than happy to review and merge if you open up a pr using docker compose.

Thanks for being open to it. I make no promises, but will noodle on a compose script this week and report back with progress.

... I'm more than happy to create a docker job in this repository that uses the distro-source image as a base image and adds and builds moveit2_tutorials to it.

That would be great - having an image that skips the VCS fetch, rosdep install, and caches the debug build means the reader can skip the whole "getting started" doc and get straight to the fun parts!

I am not very certain why this is the case but I can confirm.

Perhaps we can just point this out as an ignorable error for now? #412

Good catch! We are trying to use more moveit and less moveit2, we should perhaps change the documentation to use ws_moveit instead of ws_moveit2

Cool - #411

We should document adding apt update.

Added in #412

It changes from people to people... Can you give a few examples on what root permission issues did you face?

Got it, thanks for the clarification! I usually develop by mounting a repository from the host as a volume on the container. That benefits from a non-root Docker user as otherwise files generated by the container end up owned by root and become uneditable outside the container without running sudo. I've had no issue with using root when there are no mounted volumes.

Benefits of mounting a host volume IMO is that it's easier to add external resources (images, configs, files from other projects etc.), doesn't hinge on the life of the container (I've had to docker system prune from time to time), and there's no need to push credentials around for running git commands inside the container. The downside is extra config overhead in volume mounting and some potential weird edge cases (e.g. cross-boundary unix sockets on OSX are considered wontfix).

This is absolutely an area of preference though - there's not a "best" dev setup any more than there's a winner in the vim vs emacs argument :)

I think root gives much less learning curve using docker.

Yeah, you're probably right there. Less can be more :)

[vatanaksoytezer]

@smartin015 Thank you so much for the PRs!

[mikeferguson]

Since Debians are out now, should we change the recommendations?

[vatanaksoytezer]

Yeah, I'm happy to change that

[lorepieri8]

@smartin015 Thanks for your contribution. Ironically I tripped on your
source ~/ws_moveit2/install/setup.bash
as of now it should be

source ~/ws_moveit/install/setup.bash

[tanmayyb]

Hi, I'm not sure if the subject of this issue got properly resolved, namely:

"Why Do Humble and Rolling Tutorials Require Building MoveIt From Source?"

Although the build process is smooth, it is not speedy - thus inconvenient if you only want the tutorials. If moveit can be installed as a debian, why is building from source a requirement?

Is this:

1. a technical limitation?
2. or, did we choose to not resolve the root of this issue?
3. or, can we soon expect a debian release of the tutorials only package too?

Any comments are appreciated.
Best,