First off, Java and JavaScript are entirely separate and dissimilar languages. Firefox doesn't utilize Java at all; it only uses
JavaScript.
Second, this wasn't done to improve speed; rather, it was done to make it easier to create extensions and add-ons for usage with Firefox across all platforms. Code written in C++ must be compiled; once compiled, it can only be used on the
the platform for which it was created.
Since JavaScript is an interpreted language that runs in a browser interpreter, the same extensions may often be used with MacOS, Linux, or Windows (or anything else that Firefox runs on).
And indeed, some applications employ scripting languages for this kind of function (to allow easy customization and extension). For instance, Civilization IV's UI and game logic are totally written in Python and XML due to this, but the graphics code, which is performance-intensive, is still in C++.