Unnecessary build of library file

[mikaelpatel]

For each build the core library is created. This seems unnecessary especially if there is only a minor change to a sketch.

Improvements could be:

1. Build the core library only once with the first build of the sketch.
2. Cache core library builds with board tag so that they may be reused either when switching boards or between runs of the IDE.

Cheers!

[matthijskooijman]

Do you mean for every start of the Arduino IDE, or for every build?

I know that between runs of the Arduino IDE, everything is rebuilt normally (because builds normally happen in a temporary directory - you can set the build.path preference to save temporary files between runs of the IDE).

Within a single run of the IDE, the IDE should only rebuild the files that are actually changed. However, it could be that the generation of the .a file from the .o files happens unconditionally? I remember something like this.

As for caching with a tag, the (1.5) IDE currently writes the build options used into the build directory, so it can do a force rebuild when any build options change (it's a bit pessimistic here, though). Also, it isn't currently smart to keep multiple sets of cached builds - but frankly I don't think that a complete build takes so long that you'd want to cache multiple copies, but YMMV.

[mikaelpatel]

Below is a typical sequence when rebuilding without any changes. The "avr-ar rcs core.a xxx.o" command is repeated for all the object files even though none of them are recompiled. The build should not create a new "core.a" in this case. There is no change and it is just a waste of time.

[mikaelpatel]

I would like to suggest the following.

1. A core library cache directory is used. Each library is named according to board short name; libcore-board.a
2. The core library is built if not available in the cache directory. Typically the first time a board is selected.
3. The cache can be flushed by a command (in for instance preferences).

[matthijskooijman]

Ah, indeed, the .a file is apparently rebuilt every time. I noticed that as well at some point, we should definately fix that (I might have a look at this later. If you want to look, I think it's in the Compiler.java file).

As for your extended proposal - to keep a compiled core library per-board cached. It sounds like a solid proposal, but I'm not so sure we should want this. It would require significant changes to the current architecture to keep these cached copies around (currently the build products are stored in a temporary directory). Also, deciding when to remove the cache is a complex problem.

What's you're usecase for this? Do you think the recompile is too much when you switch from one project to another? Or are you compiling one project/sketch for multiple boards and quickly switching between them? Or something else?

@matthijskooijman

[jantje]

A core library cache directory is used. Each library is named according to board short name; libcore-board.a

My 2 cents
V1 of my arduino eclipse plugin works like this. However with the changes in arduino IDE 1.5.X (and support for Teensy) this is no longer sustainable. That is why V2 of my plugin no longer has a project for the arduino core but includes the arduino core in the project.
Why?
The menu options give compile directions to the arduino core files and as such libcore-board.a is no longer "precise enough". What is needed in 1.5.X is libcore-board.option1.option2. .... optionN.a
As the options are configurable in boards.txt this creates a big problem as to when to clean-up where the benefit (compile time savings) is relatively small.

Best regards
Jantje

[mikaelpatel]

I am considering delivering re-built core libraries for the boards supported by Cosa with the release. I would also modify the IDE build so that it uses the libraries and does not compile the core source code at all. This removed a lot of build time and the scale of Cosa does not become an obstacle.

A short comment on @jantje remark. The number of board variants has actually not changed in the IDE. It is only the menu structure to handle the options/variants of the same board. The clean up is only a problem if you allow users to "tinkle" in the core source code library. AND this should never happen if they are not core developers.

Best regards, Mikael

[matthijskooijman]

@mikaelpatel, right, so the incentive for this is the bigger size of Cosa. As I said, I don't think that the build time for the Arduino core is enough motivation for a feature like this, but if Cosa takes a lot longer to compiler, I can imagine you'd want to cache that.

Perhaps this is something to discuss on the dev mailling list? The changes needed to support something like you suggest are fairly invasive, so I think the list might be a better place for this. You'll also want to get the opinion of the Arduino dev team, since they'll have the final word (I'm just an interested contributor).

[mikaelpatel]

I have already solved this for Cosa build using traditional makefiles. It is a question about improving the Arduino IDE to scale for larger projects with more files. They could be files in an improved Arduino core with more support out if the box as needed for the new boards or files from libraries.

Anyway it seems like I need to discuss what industry quality software and build is all about first. I will close this and move the discussion to the dev mailing list.

Thanks for your effort and comments.