A Graphical Editor for the GMF Mapping Model

Everyone who has ever worked with GMF knows that  it can be painful due to the steep learning curve and the tools it provides for editing the GMF models. The complexity of the models in combination with the very basic tooling makes learning and using GMF harder than it should be. Recently, efforts have been made to improve this situation. There is for example EuGenia, which was presented at the ESE 2009. EuGenia uses annotations to add information to the domain model so that the GMF GraphModel and MapModel can be generated automatically. For my diploma thesis however, I followed a different approach to improve the GMF tooling. Jens gave me the task to create a 3D graphical editor for the GMF mapping model. I would like to present the result of my efforts in this blog entry.

The basic concept for such an editor can be summed up by the following bullet points:

• The editor should display the domain model, the GraphModel, the ToolModel and the MapModel together in a 3D view.
• The models should be visualized graphically, thus a graphical notation must be developed for each of them.
• References from the MapModel to the other models should be visualized using 3D connections.
• It should be possible to edit the MapModel entirely by dragging elements from the other models (or rather, their diagrams) to the MapModel. The user should not need to use any other methods to edit the MapModel.
• The editor should create as many elements of the MapModel automatically to assist the user.

Graphical Notations

Thus, the first step was to develop graphical notations for the domain model, the GraphModel and the MapModel. The ToolModel was excluded to limit the amount of work, and it can be added at a later stage. I began by creating 2D GEF editors that simply display the aforementioned models. Obviously, with the Ecore Tools there already is a graphical notation and a very capable editor for EMF based domain models, so no work had to be done there.

Ecore Tools Editor

The main focus in creating a graphical notation for the GraphModel was to render the figures in the figure gallery just like they would look like in the generated editor. The diagram elements defined in the model should be grouped by their type (node, connection, diagram label and compartment) and the layout should be automatic, without the need for user interaction. The following screenshot demonstrates this.

Graphical Notation for the GraphModel

Developing a graphical notation for the MapModel was a little more complex because it needs to convey more information than the notation for the GraphModel does:

• The MapModel mirrors the containment relationships of the domain model (using NodeReferences, NodeMappings and Compartments) which results in a hierarchical structure. This structure should be represented visually by nesting the figures which represent the model elements.
• The MapModel needs to display additional information like which domain element is mapped etc.
• The mappings should be represented by the figures which they map to in the GraphModel. That is, if a NodeMappping maps to a blue rectangle figure (in the GraphModel), then that NodeMapping should render itself as a blue rectangle also.

Graphical Notation for the MapModel

As you can see in the above screenshot, every model element is represented using a figure that has a gray border and title bar. These figures are nested, and those figures that represent a mapping (like NodeMapping) display the referenced figure as their child. Additional information is displayed at the bottom of some of the figures in the form of a list of named properties. CompartmentMappings are nested into the figures to which they belong and contain the child references that link to the contained domain model elements.

With all the graphical notations in place, the next step was to combine those into a 3D multi editor. In order to do that, the 2D GEF editors had to be 3D-fied first. A 3D version of the Ecore Tools already exists as a GEF3D example, so that was just reused here. The other editors could be easily 3D-fied by adapting four classes each.

Multi Editor

Combining the three viewers into a multi editor also is a pretty easy task with the tools that GEF3D already contains. There is an abstract base class (AbstractMultiEditor3D) for multi editors that allow any editor implementing the interface INestableEditor to be embedded. There are only very few things that need to be taken care of manually here, like for example making sure that all embedded editors use the same EMF ResourceSet to load their models. Once everything is in place, a multi editor is used by opening one of the models using the newly created editor and dragging the other models onto the editor window - easy! The result of this step is displayed in the following screenshot.

Multi Editor for the Mapping Model

As you can see, this version of the editor already displays references from the MapModel to the other models using 3D connections. This has been implemented by using the UnidirectEditPart3D base class that comes with GEF3D. This class allows creating connections that are created by either the source or target edit part, which is exactly what was needed in this situation, since neither the domain model nor the GraphModel are aware of any references that a MapModel might have to their elements.

Dropformations

At this point, all that is missing from the editor is the actual editing functionality. It was decided to use a model transformation language to implement the editing functions (which always modify the MapModel in some way). This transformation language is called Mitra was created by Jens von Pilgrim for his Ph.D. thesis. Mitra is an acronym that stands for "Micro Transformations". The language is optimized for fine-grained, semi-automatic model transformations, as the name indicates. Since the Mitra rules that implement the editing functionality are triggered by drag and drop operations, a GEF tool is needed that can do this. Generally speaking, a drag and drop operation can also be interpreted as selecting a number of parameters for a rule. The model elements that are dragged are called source parameters while the model element which the elements are dropped on is a target parameter.

Consider the following example: Say you want to create a LinkMapping. For this you need a connection from the GraphModel and a domain element that represents the association in the domain model (this could be either an association class or a simple reference). For the sake of simplicity, let the association be modeled by a reference in the domain model. Now to create a LinkMapping, you can drag the connection from the GraphModel and drop it on the MapModel. This creates a new LinkMapping. To set the link feature, you would now drag the association from the domain model and drop it on the LinkMapping, etc.

There is one remaining problem however: How are the transformation rules selected? Currently, whenever an element is dropped, a popup menu is displayed that contains all rules defined for the editor. This is demonstrated in the following screenshot.

Rule Selection Menu

Obviously, this is not optimal, and there are plans to implement automatic rule selection by matching the parameter types against the rule signatures, but this is still on the todo list. But even if this is implemented, there will be cases in which there are multiple rules that match the parameter types, and in those cases, the rule name should be replaced by some more meaningful names in the menu.

Demonstrations

This first video shows how a TopNodeReference (including a NodeMapping, LabelMappings and CompartmentMappings) is created simply by dragging a Node from the GraphModel onto the MapModel. 

Note that no references to the domain model are set - to see how this is done you can watch the next video:

 Finally, the last video demonstrates how to create a ChildNodeReference inside an existing CompartmentMapping and a LinkMapping:

Kristian

GEF3D tutorial article in English language available

I'm happy to announce the English version of the GEF3D tutorial article "GEF goes 3D" (source code see here), available online now at JAXenter.com. The German version has been printed in the Eclipse Magazin 6.09 and is online available, too. This article is the first part explaining the basic concepts of GEF3D and how to "3d-fy" a very simple GEF-based editor. The second part about creating multi-editors is already available in German language (I will translate it as soon as possible), a third part about how to 3D-fy GMF-based editors will probably be published in the next issue of the Eclipse Magazin. A list of all GEF3D related articles and papers can be found at the GEF3D website. Thanks to Hartmut Schlosser for publishing the article online and Madhu Samuel for proof reading the english version.

BTW: Stay informed about latest GEF3D news by following GEF3D at twitter

Jens

GEF3D in print, part 2

The GEF3D series in the German Eclipse Magazin continues. In the current issue you will find the second part explaining how to create multi-editors. The editor described in the first part is enhanced in order to display multiple graphs in a single 3D scene, additionally colored markers can be added to demonstrate inter-model-connections.

If you have missed the first part, don't worry. You can read it online at JAXENTER. A third part of this series is planned for the next issue, we will then describe how to 3D-fy GMF-based editors.

See GEF3D in Action at ESE and Around the World

The GEF3D team will present GEF3D at Eclipse Summit Europe and at several DemoCamps around the world in the next weeks. If you visit ESE 2009 and if you stay till the very end, you can attend Kristian's and Jens' talk about "GEF3D: Overview and Applications" (Thursday, 4 pm). If you miss that, don't worry! We will present GEF3D at several DemoCamps in November (and December).

Madhu Samuel will present GEF3D at the DemoCamp in Bangalore, India. Madhu has done a great job writing an GEF3D-JOGL adapter (currently, GEF3D uses LWJGL, but in the future, JOGL will be the standard OpenGL-wrapper library used by GEF3D), which is not yet publicly available (but we will add his code and JOGL to GEF3D soon).

On 23th November, Kristian will present the results of his master thesis at the DemoCamp in Berlin, Germany. His application is really cool: It enables the definition of an GMF mapping model using GEF3D, that is you don't have to fiddle with these tree-based editors anymore, instead you can use a 3D graphical editor and simply drag and drop your figures and domain elements into the mapping model.  If you have ever wondered why to use GEF3D -- his application is the answer!

I will present some GEF3D applications at the DemoCamp in Hamburg, Germany, on 4th of December (this November DemoCamp is in fact a December DemoCamp).

Looking forward to meeting you!

Huh, Hamburg seems to be an attractive place for people from Berlin. I've noticed Stephan Herrmann's presentation about "Adapting EclipseLink for Object Teams" in Hamburg, too. Object Teams (OT) is a very interesting framework for aspect oriented programming (Eclipse project proposal), and sooner or later I will have to try out to 3D-fy editors using OT :-)

Jens

GEF3D in print

In the current issue of the German Eclipse Magazin, you will find an article introducing the basic concepts of GEF3D with a short tutorial demonstrating the "3d-fication" of an GEF editor. This article is in German, an English version is planned. A second part of this article explaining how to create multi-editors and how to 3D-fy GMF-based editors will probably be published in the next issue.

Jens

GEF3D and Mac OS X 10.6

I have installed Mac OS 10.6 Snow Leopard on my development machine yesterday and I'm happy to report that Snow Leopard comes with a 32bit Java 6 VM and GEF3D works flawlessly in 10.6 in both the 32bit and 64bit VMs. Generally, I have the feeling that Eclipse is more snappy on 10.6, but that may also be because I have made a fresh install. Performance may degrade a bit due to harddisk fragmentation and other factors.

Kristian

Pick Me, Rotate Me!

There were two nasty issues in GEF3D. Actually there was no bug report on the first one, as it was no real bug but a conceptional problem we had with picking (that is selecting a 3D object in a scene with the mouse). The other one caused many strange behaviours, documented in several bug reports (280977, 262529,
263441, 278634). I'm proud to say that we (Kristian, who did the main work -- great work, Kristian! --, and I, who helped out when Kristian couldn't see the forest for the trees ;-) ) solved both issues. This is a big step toward a final release, as GEF3D now supports almost everything necessary to 3D-fy existing editors and use them in a 3D environment with extra features such as inter-diagram connections! Unfortunately, both solutions made some API changes necessary, but in the end it makes it even easier to 3D-fy existing 2D editors.


Picking


The first issue was about picking. We used color picking because I initially thought that this would be much easier and faster. In layman's terms, color picking simply means to render a scene twice: one time as it is shown on the screen, and a second time with a single unique color for each pickable object. Picking an object is as simple as getting the color at the mouse position and looking up the object with that color in a map. The advantage of this technique is that you do not need to implement any geometric calculations and that it is pretty fast and accurate.



Unfortunately it was really hard to use GEF's findFigureAt methods with this technique, because GEF allows to add a filter to that method in order to ignore certain figures. The problem with color picking is that it is impossible to find a figure that is located "behind" an ignored figure, unless the color buffer is rendered again without the ignored figure. We used some cumbersome tricks to work around that problem, but in the end we had to find a new solution.



So we decided to remove color picking and implement geometric picking. That is, we have added methods to all objects shown in a 3D scene which can calculate the intersection with a ray. In order to improve performance, every figure can now calculate its paraxial bounding box. A paraxial bounding box is simply a cuboid whose edges are parallel to the world coordinate system axes and that the figure itself and all its children (see Fig. 1). If a ray hits the paraxial bounding box, the boxes of the children are searched and so on. Finally, we determine whether the shape of a figure intersects with the ray. This way, we can fully support GEF's findFigureAt with filters (in GEF, a TreeSearch is used for filtering). As a side effect, rendering shapes is much easier, since they have to be rendered only once -- no color buffer has to be rendered any more. The trade off is that every shape now has to implement a new interface Pickable, which means it must be able to calculate the intersection of a ray with its shape. But don't worry: We provide a bunch of 3D shapes which implement this interface, and you can use these shapes to compose your own shapes. Also, we plan to provide more shapes in the future, so that in most cases you do not have to deal with these 3D issues at all.

Figure 1: Paraxial bounding boxes, rendered in Draw3D debug mode (use preferences to toggle debug mode)

Coordinate Systems and Rotation


The second issue was about coordinate systems. The problem with coordinate systems in 3D (and sometimes also in 2D) is that there are several coordinate systems, and frankly, in the end we lost control of where which coordinate system was used within GEF3D. Due to these coordinate problems rotated figures caused a lot of problems. Thus we studied and refactored the code dealing with coordinate systems. Kristian has written a wiki article about that, explaining our solution in detail. In summary, GEF3D works with three coordinate systems:

1. a world coordinate system with absolute 3D coordinates
2. mouse coordinates, that are 2D coordinates relative to the canvas
3. surface coordinates, that are 3D coordinates relative to the surface of a 3D figure. You will however notice that GEF3D often uses 2D surface coordinates, which are the projection of the X and Y component onto the plane Z=0.

2D content is projected on surfaces, and since 2D code is handling the 2D content, GEF3D needs to hide all 3D related issues from figures (and their controllers) on surfaces.
To cut a long story short, we were able to implement surfaces completely like sandboxes for 2D content. This is not an entirely new concept, as it was already possible to 3D-fy 2D content, but surfaces make that even simpler. As a result, all tools of an embedded 2D editor are now working out of the box, that is new elements can be created, moved or resized without the need to change these tools. In earlier versions, we had some problems with rotation, but even that problem is solved now. As you can see in Figure 2, you can simply move a 2D figure from one surface (1, 2) to another, the feedback figure adjust to the current surface (3,4). Of course, in order to actually move the model (of the figure), the editor has to support that kind of operation. As you can also see, the connection between two 2D nodes on different rotated surfaces is correctly located!

Figure 2: Moving a 2D figure from one surface to another.

Changed Rendering Strategy and Shape Library

Another issue not mentioned above is transparency. The problem is that OpenGL supports translucent colors, but no real transparency. That is, an object may have a translucent color, but the object is handeled by OpenGL just like an opaque object. That is, an object "behind" an transparent object is not painted if the transparent object is rendered before the object behind it. If the transparent object is rendered after the object behind it, then the colors are blended and the object is rendered transparently.

In order to implement "real" transparency with OpenGL, the programmer has to ensure that (transparent) objects are rendered after opaque objects in correct order, that is from back to front. In GEF3D, 3D figures were rendered by (recursively) calling their render method. If a figure had to be rendered tranparently, it created a temporary TransparentObject, causing the render engine of GEF3D to render the transparent objects after all opaque figures had been rendered (i.e. in a second render pass).

Unfortunately a figure may be composed of other figures, which causes a big problem. If a child figure is transparent as well, its transparency property was only be recognized when it was rendered, that is, when its (transparent) parent figure has been rendered in the second render pass. Then it was impossible to add the (transparent) child into the sorted list of transparent objects correctly, as figures of this transparent object list may have been rendered already. In order to solve that problem, we changed the overal rendering strategy in GEF3D:

Instead of (recursively) calling the render method of each 3D figure, a newly introduced method Renderable.collectRenderFragments(RenderContext renderContext) is called (recursively). Instead of rendering the figure, the figure adds so called RenderFragments to the render context via RenderContext.addRenderFragment(RenderFragment i_fragment). When all render fragments are collected, the render context firstly renders all opaque fragments, then sorts and renders all transparent fragments, and finally renders all superimposed fragments (that are objects to be rendered after anything else, e.g., in case of feedback figures). If you acutally have implemented some render code in your figure, you will have to implement the interface RenderFragment and add implement collectRenderFragments:

public void collectRenderFragments(RenderContext renderContext) {
return this;
}

In order to avoid the initial problem, RenderFragments are defined as leafs, that is they are not allowed to be composed of figures. The old interface TransparentObject has been removed.


Fortunately, in most cases you do not have to bother about these things, as you could use shapes as explained in the next section!

Shapes

Due to these changes, and especially because of the need to calculate the intersection of a figre with a (pick) ray, it becomes more interesting to use predefined shapes. Currently, Draw3D includes cuboids, spheres, cylinders, cones, truncated cones and polylines as primitives. If you look at the shapes package (org.eclipse.draw3d.shapes), you will notice that some of the shapes come in two flavours, for example there is a CuboidShape and a CuboidFigureShape. A CuboidFigureShape actually wraps a CuboidShape and adds some convenient functionality in that it automatically sets some graphical properties of the CuboidShape. For example, it sets the CuboidShape's fill and outline colors to the foreground and background color of the figure to which the CuboidFigureShape is linked. This makes it very easy to create a cuboid shape that represents a figure. Just create a CuboidFigureShape and pass the figure to the constructor and everything else is handled for your.


We plan on adding such convenience wrappers for other shapes as well as soon as they become neccessary. If you need such a wrapper, let us now (by writing a post to the GEF3D newsgroup).



In order to use a shape, the figure has to create it somewhere and then add the shape (which implements both, the RenderFragment and Pickable interfaces) in collectRenderFragments to the render context. Since this technique is used rather often, a new figure called ShapeFigure3D has been introduced. All you need to do is to simply implement its abstract method createShape() and create the shape there, e.g.,

@Override
protected Shape createShape() {
return new CuboidFigureShape(this);
}

Note that ShapeFigure3D does not only use the shape for rendering, but for calculating the distance (Figure3D#getDistance(Query)) and the paraxial bounding box (getParaxialBoundingBox(ParaxialBoundingBox)) as well! You may have a look at this figure if you want to implement you own figure from scratch!

General API Changes

The following list shows the changes which were necessary in order to adjust an editor created with an elder version of GEF3D to the latest revision (rev. 295):

1. Changes usually found in your graphical editor:
   
   1. GraphicalViewer3DImpl no longer implements IScene. Instead, LightweightSystem3D now implements IScene, so if you need the IScene, simple replace the viewer with viewer.getLightweightSystem3D()
   2. LightweightSystem3D.addRendererListener(RenderListener i_listener) was renamed to move to IScene.addSceneListener(ISceneListener i_listener)
   3. In order to show correct 3D handles and feedback figures, 2D edit parts are to be modified. This was already the case in earlier versions, it has become easier now and most cases of feedback creation are implemented. Some new policies have to be installed as follows:
      
      • Create a node: ShowLayoutFeedbackEditPolicy3D (see Fig. 3, (1) and (2))
      • Create a connection: ShowSourceFeedback3DEditPolicy (see Fig. 3, (3) and (4))
      • Move or resize a node: Handles3DEditPolicy, to be installed at parent edit part (e.g., diagram) (see Fig. 3, (5) and (6))

      

      Figure 3: Feedback in 2.5D and 3D, simply by adding some policies.

      
      
      Only selecting a connection is not implemented yet, we will fix that as soon as possible.
      
      
      The best way to install these policies is by using the borg factory pattern, e.g.,
      

      
      EditPartFactory originalFactory = getGraphicalViewer().getEditPartFactory();
      BorgEditPartFactory borgFactory = new BorgEditPartFactory(originalFactory);
      
      // replace diagram edit part
      borgFactory.addAssimilator(new EditPartReplacer(GraphEditPart.class,
      GraphEditPart3D.class));
      
      // modify diagram edit part's policies 
      borgFactory.addAssimilator(new AbstractPolicyModifier() {
      
      public boolean match(EditPart part) {
      return part instanceof DiagramEditPart3D;
      }
      
      public void modifyPolicies(EditPart io_editpart) {
      // feedback when creating a node:
      io_editpart.installEditPolicy(
      ShowLayoutFeedbackEditPolicy3D.ROLE,
      new ShowLayoutFeedbackEditPolicy3D());
      // handles and feedback when moving or resizing a node
      io_editpart.installEditPolicy(
      Handles3DEditPolicy.CHILD_DECORATOR,
      new Handles3DEditPolicy());
      }
      });
      
      // modify node edit part's policies
      borgFactory.addAssimilator(new IAssimilator.InstanceOf(
      NodeEditPart.class) {
      
      public EditPart assimilate(EditPart io_editpart) {
      // feedback when drawing a connection
      io_editpart.installEditPolicy(
      ShowSourceFeedback3DEditPolicy.ROLE,
      new ShowSourceFeedback3DEditPolicy());
      return io_editpart;
      }
      
      });
      
      getGraphicalViewer().setEditPartFactory(borgFactory);
      

      
      
   
   
2. Changes in figures / edit parts: Often, diagram figures in 3D are simply figures displaying a cube. We provide a cube shape which can be used by the figure. Since using shapes in a figure is a very common case, a new ShapeFigure3D has been introduced. Here is an example of how to use that in combination of a diagram figure which provides a surface as well. In earlier versions, surfaces were implicitly provided by each 3D figure, now you have to explicitly provide a surface. A surface is only needed if 2D content is to be projects onto the 3D figure.
   

   
   public class GraphFigure3D extends ShapeFigure3D {
   
   private ISurface m_surface = new FigureSurface(this);
   
   public GraphFigure3D() {
   SurfaceLayout.setDelegate(this, new FreeformLayout());
   getPosition3D().setLocation3D(IVector3f.NULLVEC3f);
   getPosition3D().setSize3D(new Vector3fImpl(400,300,20));
   setBackgroundColor(ColorConstants.white);
   setAlpha((byte) 0x44);
   }
   
   @Override
   public ISurface getSurface() {
   return m_surface;
   }
   
   /** 
   * {@inheritDoc}
   * @see org.eclipse.draw3d.ShapeFigure3D#createShape()
   */
   @Override
   protected Shape createShape() {
   return new CuboidFigureShape(this);
   }
   }    
   

   
   

We have adjusted all examples (the 3D ecore editor, 3D UML editor, and 3D graph editor), so you may have a look at these editors.


I have written a tutorial article explaining the basic concepts of GEF3D, it will be published in the next issue of the german journal Eclipse Magazin. An english version of this article will be made available as soon as possible. In this first article, a sample GEF editor is 3D-fied. We have planned to write a follow-up article explaining how to 3D-fy GMF based editors as well.


Jens