For full details, be sure to check out What's New in GWT 1.6?, and then download the release candidate and give it a whirl. Let us know what you think in the GWT Developer Forum.
Jason Freeman recently created Piano Etudes, an audio-enabled GWT application integrated with Flash. We're happy to say that Jason had a chance to stop by to talk about the development strategy he used to add audio to Piano Etudes. Check it out below.
As a faculty member at the Georgia Tech Center for Music Technology, I recently developed Piano Etudes, a musical application built with GWT, along with a student of mine, Akito Van Troyer. Since Piano Etudes is about music, audio was a top priority for us throughout the development process. In this post, I’d like to share what we’ve learned from our experiences with audio and GWT and explain how we implemented audio functionality.
There is currently no built-in audio support in GWT; nevertheless, it’s fairly simple to add audio to a GWT app. Several audio APIs for GWT exist in various stages of development, including Fred Sauer’s gwt-voices, CodeLathe’s GWT SoundManager, Jeffrey Miller’s gwt-sound, and the sound classes in the GWT Incubator. Most of these libraries in turn rely upon the Flash Sound API instead of the inconsistent audio support found natively in web browsers.
For Piano Etudes, we found it easier to build our own GWT classes to support the audio features we needed, instead of using one of the APIs listed above. We wrote JSNI methods to access Scott Schiller’s SoundManager2 Javascript Sound API. SoundManager2, which similarly relies upon Flash, is a clean and elegant API that Scott has been fanatically supporting and updating. A couple of the GWT audio libraries also use SoundManager2, though they only support a subset of its API.
Regardless of how you choose to implement audio in your GWT app, we've discovered a few key principles while working on Piano Etudes that you may want to consider:
Because audio support in GWT is rapidly evolving, the API you choose to use today may not be the one you use a year from now. So it’s essential to wrap your audio functionality in your own audio classes. Then if you change implementations later, you’ll only have to tweak a few classes instead of auditing your entire code base.
For Piano Etudes, we created two GWT Java classes to handle all audio functionality. These are the only classes in our code base that assume a particular audio implementation and are also the only classes that contain JSNI methods.
One class, called SoundManager, includes static JSNI methods for initializing SoundManager2 and configuring its global parameters.
For example, Soundmanager.setDebugMode() will enable or disable debugging:
Soundmanager.setDebugMode()
public native void setDebugMode(boolean b) /*-{ $wnd.soundManager.debugMode = b; }-*/;
Each instance of our other class, called Sound, represents a single audio file to be played back in GWT. This class wraps methods such as play, pause, stop, getting and setting panning, and getting and setting playback position.
It was important for us to hide SoundManager2’s implementation details from the rest of our GWT code. Both SoundManager2 and parts of our GWT application code have their own bookkeeping, so there is potential for bad cross usage if the code isn't kept separate. For instance, SoundManager2 requires that an ID string be assigned to each sound object, and it uses that ID in many of its methods. Our Sound class handles this ID string internally and privately. The constructor method generates a unique ID string for each instance of the Sound object:
public Sound(String fileName) { soundID = CustomIDGenerator.getUniqueId(); // additional initialization code }
The Sound object then uses this ID in other methods. For example:
public native void stop() /*-{ $wnd.soundManager.stop(this.@net.jasonfreeman.pianoetudes.client.sound.Sound::soundID); }-*/;
There is no need for any other application code to be aware of this ID string; they are able to use instances of Sound in a more object-oriented, implementation-neutral manner.
In Piano Etudes, the timing of audio events is important. Typically, each measure of moveable music is stored as a separate audio file, so when audio files do not play back at the correct times, there is an audible gap or jump between measures of music. Audio timing is similarly important in applications where sound effects are triggered by user actions or where multiple audio files must play back together in sync.
Neither Javascript nor Flash are famous for precise timing, so there is an upper bound on timing precision within GWT as well. But there are a few strategies that do improve precision:
Both Flash and GWT have limited capabilities when it comes to audio. You can play sounds, jump around within sounds, combine them together, query waveform data, and perform other basic tasks. Newer versions of Flash do support some sample-level manipulation of audio data, but it would be tough to use these features from JavaScript or, by extension, from GWT. (In fact, it’s tough to use these features at all in their present form.)
So what if your GWT app requires advanced audio features such as sample-accurate timing, sound synthesis, digital signal processing, or mixdowns to audio files? You could implement audio functionality in a Java applet instead of through Flash, taking advantage of a Java-based audio API such as JSyn, JavaSound, or Minim to access additional features unavailable in GWT. But if you’re like me, the headaches associated with Java applets are one of the reasons you’re using GWT in the first place, so that is not an enticing solution.
Fortunately, there is another alternative: render your audio on the server. This successfully avoids the limitations of client-side systems, though it does introduce other issues related to scalability, latency, and bandwidth. While it’s not a universal solution, it is a valuable strategy for your toolbox.
Here’s a simple use case: in Piano Etudes, we wanted users to be able to download the music they created as an MP3 file. In GWT and Flash there is no obvious way to render an audio file to disk or to convert it into an MP3. So instead, we handle this task server side.
When a user clicks a button to download the MP3 of their music, GWT sends a representation of that music to the server; for each audio track, it describes the audio clips in the track and the timings of those clips. We post the data as JSON via HTTP to a PHP script, but this could be done just as easily via any other client-server communication paradigm supported in GWT. Here’s some code for the client side:
FormPanel mp3Form = new FormPanel("_blank"); HorizontalPanel panel = new HorizontalPanel(); Hidden mp3Data = new Hidden("data", timeline.getAsJSON()); panel.add(mp3Data); mp3Form.setWidget(panel); mp3Form.setAction(URL.encode(SERVER_SCRIPT_URL)); mp3Form.setMethod(FormPanel.METHOD_POST);
This code snippet submits a JSON array to the server representing the audio clips and timings for each track of music. The JSON data is sent as a hidden field in an HTML form. The result obtained from the server (the MP3 file) is returned to a new web browser window, keeping the GWT app open in the current window.
When the server-side script receives the data from the client, it processes that data to create and run a simple script in a computer-music language called RTCmix. (Classic scripting languages for computer music, such as CSound and RTCmix, are great for rendering audio server-side.) The audio file output by the RTCmix script is then converted to an MP3 using LAME, and that file is returned to the client browser, which downloads it. Here is a simplified excerpt from our PHP script (our entire script is only 100 lines of code):
// initialize the rtcmix script $audio = tempnam("/tmp", "PianoEtude"); // output file $script = "set_option(\"AUDIO_OFF\", \"CLOBBER_ON\")\n"; // non-real-time $script .= "rtsetparams(44100, 2)\n"; // sampling rate and number of output channels $script .= "rtoutput(\"$audio\")\n"; // output to temp file $script .= "load(\"" . $CMIX_LIB_PATH . "\")\n"; // library path for rtcmix // create an rtcmix command to place each audio clip $timeline = json_decode($data); foreach ($timeline as $track) { $insertPoint = 0; foreach ($track as $soundObject) { $soFileName = $soundObject[0]; $soDur = $soundObject[1]; $script .= "rtinput(\"$soFileName\")\n"; // insert the audio clip into the output audio file $script .= "STEREO($insertPoint, 0, DUR(), 0.5, 0.5, 0.5)\n"; $insertPoint += $soDur; } } // execute the rtcmix script $scriptFile = tempnam("/tmp", "script"); $handle = fopen($scriptFile, 'w'); fwrite($handle, $script); fclose($handle); $cmd = "$rtcmix < $scriptFile"; $result = shell_exec($cmd); // convert to MP3 $mp3File = tempnam("/tmp", "mp3"); $cmd = "$lame $audio $mp3File"; $result = shell_exec($cmd); // return result readfile($mp3File);
You can also use client-side and server-side audio together. For example: instead of just having the web browser download this server-generated MP3 file to disk, our GWT app could use a client-side audio API to play it back inside of the app.
By using a variety of GWT or Javascript APIs to connect with the Flash Sound API, you can add rich audio features to your GWT apps; for more advanced functionality, GWT apps can communicate with a server to render audio remotely. Regardless of the exact implementation(s) you choose, keeping your GWT app optimized and your audio code structurally isolated will ensure that audio performs well in your app and can incorporate future improvements to audio support in the browser
A couple of months ago, we put the spotlight on the GWT community to highlight some of the great things that have been going on. Since then, there has been a lot of new activity to talk about, so we'd like to share these announcements with you below:
GWT Server Library (GWT-SL) 0.1.5b Released: The GWT-SL library provides developers with a way to easily use GWT with the Spring framework. In this latest release, the library provides support for integration with Gilead (formerly known as hibernate4gwt) and has been updated to work with GWT 1.5.3, and also includes many other new features and improvements.
Gilead 1.2.1 Released: Gilead lets you export persistent entites from the JVM to the outside world. For GWT, it means easy integration with Hibernate. This maintenance releases includes support for Maven support and comet4gwt among other new enhancements. If you use Hibernate, or any other of the new technologies Gilead now supports, check out the library on their project page.
Dependency Injection in GWT: You may already be familiar with GIN (GWT INjection), a dependency injection library that integrates GWT with the Guice framework. However, in case you wanted to try a new flavor of guice (pun intended), you may want to take a look at the recently introduced suco library, which offers its own guice-style dependency injection technique.
emite 0.4.6 Released: emite is an open-source library that implements an XMPP communication protocol and provides hooks for GWT integration. This means you can now make your GWT applications more chatty using this library. If you want to have your users chatting over and about your GWT application, this library might be worth checking out.
More fun stuff
Google I/O: Google's developer conference is coming back again this year at the Moscone Center in San Francisco, California. If you attended last year, you'll know that lots of great talks and impromptu chats with Google engineers await. Like last year, the conference will feature a handful of awesome GWT related talks, not to mention members of the GWT team onsite to answer all your questions. Check out the registration page if you'd like to meet some of us in person.
Community meetups are a great way to meet new people with similar passions and interests as your own. It's also a great place for the community to exchange ideas. Often you get an interesting presentation to boot.
In fact, for those of you interested and within reasonable traveling distance, the Silicon Valley Web Builders and the Silicon Valley JavaScript Meetup Group have organized an exciting AJAX Panel next week on February 18th 2009. There will be representatives there from YUI, Dojo, MooTools, JQuery and of course GWT. There's a lot of interest, so be sure to sign up before the event fills up.
I've also been talking with the SV-GTUG (Silicon Valley - Google Technology User Group) organizers about doing a GWT presentation at an upcoming meeting. We don't have a firm date yet, but I'd like to start collecting your questions and suggestions for material you'd like to see covered. You can vote on other people's ideas or post your own questions here.
Finally, if you know about or organize a local user group that has developers interested in GWT, let me know. I'd like to hear about your group, and perhaps there's a way we can help make your meetings even more compelling.
By Eric Ayers, Google Web Toolkit Team
Another round of updates is ready for the Google API Libraries for Google Web Toolkit (GWT). This update brings bug fixes and new features to three of the existing libraries. The Google Visualization API Library for GWT moves from a release candidate to a public release with a few bug fixes. The Gadgets API Library for GWT now supports GadgetAds , gadget based internationalization and an important bug fix for Windows developers using GWT hosted mode. The Google Maps API Library for GWT now includes support for the GAdsManager and reverse geocoding support. We are also announcing a major update to the Gears API Library for GWT to support the Gears 0.4 feature set, which includes Geolocation, HTTP Request, Desktop and Blob support.
This release is exciting to me not just for the new features that have been added, but also for the number of new contributors to the libraries from the open source community. Special thanks to Mark Renouf for contributing Gears 0.4 support, and all those that filed issues and contributed patches.
You can find the new libraries ready for download on Google Code.
I'm a huge fan of GWT and have been since its public release in 2006. I have spent a fair amount of my spare time working with GWT by participating in the user forum and contributors forum, submitting patches to the issue tracker and maintaining open source libraries for GWT, including ones providing support for Drag and Drop (gwt-dnd), Logging (gwt-log) and sound (gwt-voices). GWT has gone through a number of exciting releases, with 1.5 easily being the most impressive release yet. You probably know that the team is fiercely working on the upcoming 1.6 release. Andrew posted a month ago about the new features slated for 1.6. As usual, I'm not sure if I'm more excited about what's in the upcoming release or all the possible things we can do next!
I'm pleased to announce that I have joined the Developer Relations team at Google as a Developer Advocate for GWT. Some of you may be wondering what a Developer Advocate does. The role has two parts. We are advocates for you, the developer, which is to make your voice heard within the product and engineering teams at Google. We also advocate broader use of Google Developer Products and APIs. Both roles work hand in hand. By listening to developer's needs I will be advocating to improve GWT as a product. A better product in turn helps increase adoption thereby growing the GWT community.
I look forward to spreading the word and to helping make GWT even better.
