DESIGN_NOTES__14_Jn_22.txt

Working on getting visual of a POP syntax graph

-] Have Famo.us, which has hierarchical drawning structure that fits well to what had in mind for POP

-] Famo.us has animations and physics engine, so as user modifies things, they can fly around in pleasing ways.

-] Famo.us structure is almost one-to-one with the visualizer's internal structure that I was imagining..  seems wasteful to duplicate..

-] However..  need the visualization to be independent of any particular drawing facility..  for example, the visualization should be reusable with a Java based canvas that is painted via Swing, as well as via Famo.us and via C++ drawing code..

-] The visualization must be specific to the data format..  therefore, it should be part of the holder that the data is inside of.  Don't want case where new syntax is added, and have to go around to all the different places, like Java app, and javascript displayer, and so on, and change code in all those! (even if they all take a plugin, that plugin will have to be in terms of the language -- java, javascript, C++, whatever -- and the plugin will have to spit out some standard thing anyway!)  
Rather, want the visualizer to be written once, in POP, and plugged into the data-holder.  It spits out standardized visual elements..  any technology can then take those visual elements and paint them to a screen.
The visualizer has fingers directly on the raw data, in whatever the internal format is for the holder..  A holder can have a different internal format vs external format..

(update: fighting with text -- haven't found any general way of calculating the width of text, given the font..  entry after entry from google search has variations on applying the text to a canvas, causing it to render, then getting the bounding box that was generated during render..  there doesn't seem to be, even for SVG, any way to predict the bounding box!!!?!?!?!  So.. there is a cascade effect..  the text must be contained within something, like a rectangle or elipse or whatever..  and, as worked out during the rest of this design session, the visualizer is responsible for calculating the placement of all the SVG shapes..  then the visualizer MUST know the size of the text, in order to know the size of the shape, in order to calculate a reasonable placement of shapes..  
   it is tempting to leave all the shape placements specified as relative, and let the final values be figured out inside the Display..  this would also allow things like employing the Famous physics engine to calculate placements, even let the placements evolve by themselves (like the D3 examples with the graph nodes flying around to find their own placement).. that would happen without the visualizer setting the placements..  rather, the visualizer would establish the visual "connectivity".. noooo..  that D3 thing relied upon knowing that it was a graph and that arcs acted like springs and nodes acted like weights, and nodes exerted n-body attraction..  but that is all specific to graphs..  would it generalize to arbitrary syntax?  Hmmmm..  if had an "atomic" level, where inside an atomic unit nothing moves relative,  then, yes, can treat the connections between these atomic things as springs, and each atomic thing has attractive mass and inertia..  can let them run around and find a stable configuration..
   Okay, so two issues: what about connections that go off-screen to out-side-the-view portions of syntax graph, and what about the user wanting to pin particular things in particular places..  often what a feeling of solidity, reliable placement so you know where everything is..
   Right, so this: for now, visualizer calculates fixed positions, but have in back of mind keeping ability for multiple kinds of visualizer within the same srcHolder..  and visualizer has ability to expect things of the Display, such as providing physics, and visualizer can maybe ask Display what kinds of visualization list it understands..  so then can add, say, dynamic view hierarchy, where visualizer specifies things like masses and springs and lets Display move them around as it sees fit..  can evolve standard view lists over time..  the Commander will have to include some facility for not allowing a user to select a visualizer that is unable to generate any of the list types acceptable to the Display the user is interacting with.)

-] A holder may also have other data-format-and-semantics specific things, such as a querier -- the querier defines the format of queries it accepts, and those queries are tailored to the nature of the data in the holder.  Hence, say, a video holder might have a querier plugged in that accepts canonical shapes, and then searches the raw video data for frames that contain that shape.  The querier would spit out things in a format that it defines.  For example, it may have a data structure for a video frame, and spit out a stream of frames that contain the requested shape.

-] Data formats should be visually defined.. for example, I should be able to draw a picture of a graph node, and that picture is the code that defines the data structure, which can then be used in other places in the code.  For example, in the Hamiltonian Path code, could have included a picture of a graph node, and connected a line to the portion being accessed, say, in a test, or in the "for all" construct..

-] Seeing a hierarchy of holders..  so a higher level holder could have multiple lower level holders inside of it..  getting a bit awkward, say with combining queriers and visualizers -- how do you combine visualizations from siblings?  there's something fundamental here.. 

-] No data may be warehoused outside a holder..  the holder collects the things needed to interact with the data, such as modifier, visualizer, querier, and not sure yet what all else..

-] Data outside a holder must be in the form of a data structure instance, where the structure was defined by the holder the data came from.  The holder took whatever internal format the data resided in and placed it into the data structure form.  All data within a POP system that is not inside a holder must be in the format of a structure defined using POP primitives.  NO RAW BYTES!  Bit twidling only upon bit-vector structures.  An adaptor is necessary in order to get a more primitive format of data (IE, can always do conditional tests on bits held in a bit-vector format, and build up values within a higher level format structure based on the test results.. example, do the math to accumulate individual bit values from a bit-vector, but accumulate them into an integer format.  Or, can test bits that represent graph edges, and build up a graph data structure.  aso..)

-] A visualizer can include animations as visual elements that it spits out.

-] Famous will generate events that are gestures such as mouse clicks.  The event handler will use the event data to connect the event to one of the original visual elements that was visualized.  It will construct a structure that states the visual element and the gesture details, and send that to the commander.

-] the commander will convert the visual element plus gesture into some action, such as moving the insertion point, or zooming the view, or adding a syntax node..  the command sends the command to the appropriate place -- could be the drawing engine (for zoom or pan actions).. or could be the modifier for insertion point or syntax element actions..  or could be some entity that carries out the action. For example when viewing a worksheet, a gesture could trigger re-calculation, so the action would be a "perform calculation" sent to one or more processors that are connected to the worksheet.

-] The modifier has access to the holder's internal data format, and can manipulate it directly.  It receives commands, such as generated by gestures performed on the display.  

-] A worksheet lives inside a holder as well..  the worksheet can talk to processors..  for example, the visualizer may simply include a reference to a processor that spits out data, and sends the reference in the visual element.  An image may be a good example..  the image is on the worksheet for some reason..  the visualizer embeds reference to image in an element and sends that to the display.  The display then directly gets the image data and paints it.

-] Right, so one way to combine visualizers in a hierarchy, is for higher level visualizers to specify a lower-level one that is to fill in the internals of a given bounding box.  So the lower-level is sent the bounding box, and it generates a view hierarchy that fits inside of that.  Not sure whether the Display should be the one that sends the requests and integrates the various responses, or the Visualizer should send the requests and integrate the responses.


==============================================
=         Display   Design  
=
==============================================
So, question..  what do display list elements look like?  Does visualizer produce the layout, for example, specify a bounding box and position for each element, including text, lines, etc?  Does it produce the bezier curves in SVG that connect syntax graph elements?

Or, does it send something with fewer details?  Let the Display calculate exact placement of visual elements?

What about pan and zoom?  Does Visualizer handle that, or Display..  pan means moving some elements out of the painted canvas and moving others in..  zoom does too, in addition to calculating new transforms for each element painted..  So, if Display does this, then it must either have all the visual elements in it, or else it much send a request to Visualizer that triggers sending what's needed..

Soooo..  make a two-way path between Display and Visualizer, in order to handle pan, zoom, 3D rotates, and so forth?

Orrrr..  leave that to the MVDM loop..  the pan/zoom/rotate gestures go into the command-maker, and thence into the visualizer, which then sends view update..

sooo..  this view update stuff..  does Display keep a full representation of the visual elements, and then translate that into Famo.us elements?  Or, does it keep, say, a current visualizer list..  where visualizer updates simply modify the previous list sent..  then Display translates the updated list into Famo.us elements..  given that Famo.us doesn't support removing render tree elements, that might be a good approach..  and it simplifies the thing about how to figure out what a visualizer update is and what to do with it inside the Display.

Okay.. now..  just need to figure out what the Visualizer actually sends..  how much detail..

You know what?  I don't want to think about it!  Just going to do something, see what happens, and then can modify later..  probably will never get modified, these choices have a way of sticking, due to momentum of volume of work and low improvement in changing..  but, don't want to stall myself, need to keep moving..

sooooo...  something has to calculate placement of bounding boxes, and calculate translations and sizing..  and move things in and out based on zoom, pan, rotate..  and trigger redraws and resends of display data for graphs and calculation results..  seems like if put placement and transform calcs inside Display then are making the Display an active thing, giving it intelligence and control..  then, run into problems when have a view that is composed of pieces from many different holders and different processors..  for example, displaying a worksheet, where the worksheet itself is in a holder that generates a view, and locations on the worksheet are calculated by live processors, and locations are graphs which are generated by graphing processors that take calculations from other processors..  want the worksheet holder to handle generating the view, and collecting the data from the other processors..  don't want the Display to have any kind of intelligence related to this!  

Hmmm..  so, if the worksheet holder is doing all of the collecting of data and triggering re-calcs, and melding together into a view..  then Display is simply passively painting that and collecting gestures that it sends back..  the holder then knows what to do with the gestures performed upon view elements..  

So that means that the Display should not be calculating pan and zoom and rotate!!  Those are responses to gestures, and those trigger behaviors from the visualizer (which may, in turn, have to trigger behaviors from other processors in order to get updated calcs to put into the view)

Got it, so as tempting as it is to peel off such gestures in the Display and build intelligence into there..  it will cause problems..  so best to have all the visual related processing be done inside the visualizers..  

so..  means the visualizer should send out things pretty close to renderable..

Now.  The question comes up..  what format?  Does the visualizer send out SVG?  Is that universal enough?  Will that cause problems with non-browser based approaches?  If not SVG, then what?  Will need to represent paths..  does visualizer calculate Bezier curves and send those?  Or does it just say "here are end points, connect these with a smooth path"?  Does the visualizer calculate how to route arcs within a graph around the other nodes in an optimal way?  If the user supplies helpers by moving things, does the Display remember that, or the Visualizer?  

Ahhhh..  okay..  if want the user to be able to affect the visual arrangement, then need to remember the gestures they give that are related to visual arrangement..  for example, they grab bezier control points and move them, in order to affect a curve, or add new points within a path in order to route it..  those things have to be remembered!  The Display has no place to store such things!!

Right..  that means that the holder must store info related to viewing the holder contents!  sooooo..  the visualizer must be the thing that generates any details that a person may want to adjust.

Right.  Any visual arrangement that a person might want to adjust must be stored inside the holder, and so the visualizer must be the thing that generates that level of detail.  Right.  Got it.



//========================  Visualizer Conclusions  ======================
//
//
//========================================================================
So.  Yes.  That means that the visualizer must send out bounding boxes, translations, sizing, path points, control points, and so forth.  So.  Yes.  SVG is looking like a good way to go..  has to be something!  Can invent my own equivalent, which specifies paths, shapes, widths, and so forth..  but what is the gain?  The only potential is sending in binary, to save the overhead of SVG's text embodiment.  The savings worth the effort?  NO!

Okay then.  SVG.  That is the visualizer's format for POP.  Or perhaps it just sends the values that go into the SVG?  The Display does the final conversion to text-based SVG format?  Hmmmm, what about custom syntax, which is SVG from the programmer?  Just send indication of SVG elements, together with the values?  Maybe enums for the SVG indicators, or something..

Now.  Question is, how much effort?  Is it more effort to send a representation, then construct text-repr inside Display?  Or more effort to construct SVG text in visualizer and send it ready-made?

Seems easier to just construct SVG text inside visualizer and send that.. 

Heck, if send representation, then are already forcing SVG!  It doesn't constrain the Display if send the text form..  the display is just as free to convert the binary form to something else as it is to convert the text form to something else..  so if Display is not browser, and doesn't have SVG drawing primitives, it has to do the extra step of parsing SVG in order to extract the values from the text, and then generate whatever internal form it has, such as calculating paths or even pixels itself..  the extraction from text does not seem like a terribly large burden compared to the rest.

Buuuut..  don't really care!  Going to generate text form inside visualizer, because, why not?  Then the Display only translates displayList elements into its internal thing that paints.  In the case of browser, Display translates display list into famo.us elements, placing the SVG verbatim onto famous surfaces, and makes the famous surface size the same as the SVG bounding box.

In fact, display list is going to represent almost verbatim what a Famo.us render tree will look like!  It is a fairly universal declaration..  hierarchy of local visual contexts, with relationship among those contexts, including relative placement in 3D space and relative sizing from one to another.

Alright, getting to it now..  display list will be a serialization of a hierarchy..  The top of the hierarchy is the view painted onto.  The Display tells the visualizer what the pixel dimensions are of the view.  The visualizer makes that the top level bounding box.  It then inserts bounding boxes within that, and inserts bounding boxes within those, and so forth down.  Each bounding box is either a hierarchy element, relationship element, or a painted element.  Hierarchy elements simply state what elements are included, and give their own bounding box.  Relationship elements are attached by the parent hierarchy element.  For example, it is the job of the parent to make sure all the children fit inside its bounding box.  The bounding box may be 2D or 3D.  A parent ensures the fit by attaching relationships to the children elements.  The relationships can be translations or scalings or rotations.. or actually any arbitrary 3D transform.  A hierarchy element knows nothing of the context in which it fits, so it has no way of attaching relationships to itself!  Each hierarchy element is responsible for ensuring that all descendants fit within its bounding box.  Any relationship attached to a hierarchy element is transitively applied to all descendants of the hierarchy element.  For example, if one is rotated in 3D, then it, together with all its contents, is rotated as a unit.  Each descendant maintains the same relationship to its parent as before..  but the entire bundle has been rotated as a unit!

Great..  so that is the display list..  it is a serialization of that hierarchy.  A display update then assumes a given hierarchy, and says "change this relationship between this parent and this child" or "remove this parent and all descendants" etc..  in effect, it specifies a change to the hierarchical structure..

Sooooo..  means that have two copies of the view hierarchy..  one inside the visualizer, which is serialized, and another inside the Display, which is translated into Display-local elements that are then painted.

Not the most efficient..  hurts me..  but, well, what are you going to do?  What's better?  Need the flexibility of having the visualizer control all changes to the view hierarchy.. and need the visualizer to calculate all placement and path details so that user changes can be remembered inside the holder (without something clunky like "save this view modification info as a black box and hand it to the Display, which knows what it means" yuck, just begging for loss of sync when move from one Display to another or even upgrade the Display!)..  

so, then, there's no way around having the visualizer internally represent the bounding box hierarchy as it constructs the details.  Thennnnn..  something must translate that representation into the internal Display representation!  The Display could have any native form internally..  canvas with paths drawn upon it, Java 2D, even just pixels, with all the code in between..  well, in effect, every Display ends up being everything between that hierarchy and on down to pixels!!

So, could just hand the data structure from visualizer to the Display..?  This is fine if both are inside same hardware, sharing memory..  but need a serialization if Display has separate memory..  which it will if the src holder is on a server and the display is a browser on a client!

Hmmm..  what about just handing the data structure to the Display for now..  can always add the serialization later, it's not functional!  :-)  Ahhh, yes.. less work.  Good.  Okay, will do that.  save serialization for later.  For now, in javascript, will build a src holder that saves syntax graph natively in JSON, as an internal data structure of the form made for GabePatterns.  The visualizer in the src holder will generate a view hierarchy from that data structure.  It will save any view related data within the syntax graph data structure as extra properties added to the nodes.  The visualizer will then hand a reference to the view hierarchy directly to the display object.  The display object will then generate famous elements from the view hierarcy elements.  View updates will simply be the visualizer changing the view hierarchy then calling the display with a reference to the changed hierarchy.

Done.  Make it so.

BTW, thinking to introduce another component, which is a helper for developers making custom syntax, that maybe takes some higher level view form and generates the view hierarchy from that..  this will act as a library for them, essentially..  (and, of course, there will be a serializer that turns a view hierarchy into a stream of visual objects sent to the Display, which will reconstruct the view hierarchy then translate that into elements that it paints.. for now, that serializer is just JSON!  It has provision to translate javascript objects into text form, and then on the other side parse that text form to reproduce the javascript objects.. might need extra logic in order to recreate the links among parents and children.. not sure whether JSON can go deep that way..).

//======================================================
//					Code Reuse
//
//======================================================

Actually, there are multiple kinds of code re-use:
-] in-line code, where have something that assumes a context (such as variables in the scope), and grabs what matches within that context and inserts that into the respective places inside its body.  For example, a macro does this, so do in-line functions.
-] processors that are communicated with.  Many function calls are this -- other places in the code are other processors, that send parameters to the function, and receive back a return response.
-] inheritance, where have a namespace hierarchy that is searched.. the context can be assigned to a variable, then a function name can be looked up within it (assigning an object to a variable equals assigning a namespace to a variable).  If the function name isn't found in the immediate context, then the parent(s) of that context is(are) searched, until the name is found.  So, code is re-used by making it accessible from a parent name-space.

In POP, in-line code has four forms:
-] meta-agents inside the syntax-graph/MVDM-loop, which contain code that is used as part of generating code that is inserted into the syntax graph.  For example, active syntax generates code at the point that the user gives the gesture to add it to the syntax graph.
-] selecting a hierarchy unit to be deep-copied to the current hole in the syntax graph (the syntax-graph of the original is duplicated)
-] selecting a hierarchy unit to be reference-inserted into the hole (the syntax graph of the original is hooked to, not duplicated)
-] macros -- in POP these are both meta-agents and also deep copies.  The macro selects things from the code surrounding where it is placed, and sucks those inside itself, and puts in a copy thus modified, into the syntax graph.

In POP, processor-communicated-with code has at least two forms:
-] connecting the inputs and outputs of one code-processor to another (this can be implied, for example on a worksheet, or direct, using the MVDM loop while in System (OS) view)
-] when sending a context to a processor-box, it creates a new processor to handle the incoming context.  That is equivalent to copies of the code at runtime, talking to each other -- the creation as well as the contents is communication..  but it's not like other forms, where inputs are connected to outputs!

In POP, inheritance can be accomplished via the copy-by-reference mechanism, or via the send-a-namespace/context mechanism (simply insert the variable connected to the processor that you want to receive sent data.. same variable name in each context, connected to different processors == polymorphism..  and the parent context attached, that has functions attached to names in it == inheritance)

In effect, POP has functional, object oriented, procedural, and declarative styles that can be done within it.  At least, the underlying mechanisms that set those things apart are all available within POP.

==============================================================
==
==       Editor and Gesture and Commander and Modifier
==
==============================================================

In famo.us, seeing putting up a rectangle that is placed in front of the others, and always in the plane of the camera, as the "menu"..

always have a set of mappings to the keys -- these mappings are set by the commander inside the srcHolder -- they are specific to the kind of data being visualized.  In the case of code, for the moment, only visualizing the code in the form of a syntax graph (will do the custom syntax visualizer next -- it will have challenges don't want to get into just yet -- need a way to create syntax graphs and turn Gabe loose on search and match to patterns, so minimum path to that).

So, seeing moving cursor around, and when hit a key, say the 'e' key, the port closest to the cursor will sprout an arrow that ends at the cursor..  the cursor is then moved to the target port and hitting e again locks onto there.. or perhaps just a left click, or perhaps space bar, etc..  choose in practice the one that feels most intuitive and convenient.

When cursor is over empty background, hit a key, say the 'a' key, and that creates a new empty element box..  s creates a new empty properties box..  space bar brings up the menu that shows the key mappings.. or maybe ctl-space..

another key zooms in on the elem or property box under the cursor

inside the box, can hit a key to move cursor between fields -- at each field, can use space to pop up another menu and the un-pop it..  which has the starting sequences of the valid entries for the field (IE, a list of the elem types -- might itself be a hierarchical menu -- pick one of the keys or sequences from this menu, then that pops up another menu that has its own sequences or keys..  eventually a leaf is a value placed into that field) --  or the menu entry describes the format of what can be typed into the field (Ex, variable name format)

===========
Soooo..  to make this work..
===========

Each bounding box in famous has an event handler on it..  want the first enclosing bounding box to receive the event.  The bounding boxes must have parent links so that can navigate out to the boxes that enclose the target box.

the event handler is fixed and generic to the Display..  however, the gestures understood are in an array that is given to the Display by the Commander.  When Display receives the "connect", it in turn sends a connect back to the Holder.  This asks for the Commander to be given to the Display.  It then tells the commander to give its gesture-handler pairs.

In javascript, the "gesture-handler pairs" is an object that has a field for each gesture it understands, and the value of that field is the function that does the computation related to the gesture.  The function receives the event object, plus the smallest enclosing bounding box.

The gesture-handler then turns the event plus bounding box into a command.  For example, if previously a gesture caused the start of the mode for typing text into a field -- say for entering the name of a variable -- then the gesture handler looks that up, within state enclosed with the gesture handler but shareable among different handlers.  The code then branches to where it extracts the key value from the event object.  It reuses a "key into field" command object, and packs it with data.  The data includes the model-object that is attached to the appropriate bounding box (the handler does the work of figuring out which bounding box corresponds to the syntax graph object that owns the field that is being typed into).  So, each bounding box has a back-link to the Model object that the shape inside the bounding box represents, where appropriate (IE, the bounding box for text has no back-link, but the bounding box for the element-box shape does have a back-link to the element object!)  Might make it just "useful info" -- for example, a text bounding box might include the name of the field of the object that the larger enclosing bounding box back-links to.  The gesture-handler will also extract any needed info from the event object, such as which key was pressed, and add that to the command-object.

The command is then sent to the Modifier, which changes the back-linked-to syntax graph objects.  As part of that, it also modifies the visualized-sub-graph, and notifies the visualizer to update.

=====================================
Okay, have a view-tree cross-linked to the basic syntax graph..  and it's working, being painted by the Display.

Now, need to get a JSON dump of the syntax graph, store that on disk, and then get it back in, as objects in javascript.

So, JSON does stringify for dumping, and parse from bringing it back in, and they even allow overrides and "translate/restore" functions for difficult values..

The one issue is that they don't handle circular data structures, and the syntax graph has to be circular..  because a gesture can land anywhere in it, so have to be able to start at any point and navigate outward..  so ports have to be bi-directionally linked to each other, which is a circle.

Now..  how to deal with that..  what approach to traverse the syntax graph, parse the pieces out as JSON, and then to reverse that..?

I'm thinking just do a standard spanning tree algorithm, and turn all port links into IDs..  When push out.. overwrite, then do JSON stringify, then replace the links..  (means need exclusive access -- can't be sharing with others..)

==
So, the steps:
-] start at root, get first element..
-] it will have properties attached..
-] it will have an array of ports in and array of ports out
- -] these ports will have pointers back to the element.. those must be replaced with ID before stringify
-] it will have array of linked elements..  those can potentially link back around in a circle..  replace those links with IDs before stringify
-] visit each port
- -] each may have pointers to paired ports..  replace these with IDs

after that, can safely stringify the element, which will catch all the ports, and all the element properties and all the port properties

Then, find the next element to visit..  do that by:
-] visit all linkedElems
-] visit all ports, and for each port, visit all paired ports, and for each of those, visit the linked elem.
-] before visit an elem, save position are in -- on stack or in struct
-] When visit elem, check whether marked as already visited.  If yes, restore position saved, if not, process as above.
-] when all reachable neighbors of elem done, restore position saved before visiting elem.  If none, then are done.

==
Now, have view boxes attached to the elements (there is a tree of viewboxes for each element, which includes boxes for the ports ((which only provide scaling and translation of the sub-graph that appears inside the port)) and any visual cues caused by properties or derived from properties)

disconnect the ones for this elem from the ones for other elems..  so a given box can only be reached from exactly one elem!

Soooo..  what need is to have two separate kinds of hierarchy..  there is hierarchy for a single element node, which includes all of the things that are visualized for that element, including boxes for empty ports, and indicators of the boundary around what should be drawn inside the POSITION of a given port.  Call that an view-set.

Then there is hierarchy between different view-sets..  for example, which other element goes inside a given port's boundary box.. this is what applies parental shift and scale transforms.
   So there are some cases here..  element inside port of another element, and element that is a linked element of another.  Also case of ordered children of a common parent.
   For example, in the Gabe Transform rule, the top level element is a hierarchy element.  It has a big blue arrow drawn between the query pattern sub-graph and the resultant pattern sub-graph..  the tail goes to the parent's first child, the tip to the second child, and the two children are arranged side-by-side inside of the bounding box of the parent..  where the parent has no shape to represent it directly, but rather a positioning of children rule and the blue arrow between children.
   Hence, that top level element has, as it's view set, a view box that has the blue arrow as its shape, and has two transforms, one to be applied to each child that puts the child in place and scales it.
   
So, it is up to the parent to decide position of each child hierarchy..  has to know the contents of the child hierarchies in order to calc the size of bounding box..  but, will deal with sizing later..  rely upon user interaction to get the sizing right..  do sizing as graph grows, one bit at a time..  for now, the parent has a translation and scale to apply to the child, while the child has its own declared size..  so, that is view-set to view-set

So, each view-set has rules about how to arrange things inside the set..  and another set of rules about how to arrange children or neighbor sets!

So, the visualizer is the thing that knows the rules and calculates placement and size (scale)..  this results in artefacts: links between boxes and links between sets, and info on translate and scale!

so within a set there is no scaling.

between sets, either of them can state itself as reference for the pair, and then state a translation and scaling for the other.  It is an error for two directly linked to both state themselves the reference for the other.  However, a given set can act in one case as a reference and in another as a recipient, across different sets that are paired to it.

So, it is the link from one view-set to another where the reference designation lies, along with the translation and scaling of the recipient

Hmmmmmm..  means that a link between two viewbox-sets is a full object rather than just a pointer!

Hmmmmmm..  want to specialize to the point that encode this distinction about view-set versus simple viewbox?  What about making it a view-set box..  leaving out any notion of element?  Then it just says that view boxes can be grouped into sets that are treated as a single whole, as far as translation and scaling are concerned..

Then, it is the visualizer that recognizes what gets put together as a view-set, and calculates the translations and scalings that one view-set imposes on another view-set..

Take an example: Summation symbol..  the body is a separate view-set, but the summation view-set determines the scaling and position of the body set.  

   There needs to be some connection, from the transform that places the child, to the root view box of that child.  The child has its own root element, and that element has its own view set.  So, that view set should be attached as the contents of the child-position in the parent -- the root view box of the view set should be attached to the end of the chain of child's transforms in the parent.
   Do this connection inside the link object that links two view-sets.  The link states which is the parent, and it states a translation and scaling to be applied to the child view set.  The translation and scaling is relative to the parent's origin (which can, in turn be a child for some other view set, such that it was translated and scaled, and so on..)

====================
Implementation:
-] create a view-set object
-] two kinds of links in the view hierarchy.. 
- -] the view set acts as root/head to its contained viewboxes, which form a tree.
- -] the view set has array of objects that act as links to other view sets.  The link objects are shared by both view-sets, and back-link to the view sets.  

Hence, a view-set determines the relative positions and sizes of other view sets via the link objects.  each view set knows only about the link objects it is connected to.

-] view set link object has two pointers in it, one to the parent view-set and one to the child view-set.  A link object contains an offset and a scale, that are relative to the parent view-set's root viewbox, and are applied to the child view-set's whole view tree.


//==============================================================
//==
//==   Menu, Commands, Interaction -- assigning responsibilities
//==
//==============================================================

-] Something has to generate the contents of a menu -- that is specific to the commands that are valid, and relies upon a context.

-] Something has to receive the gesture that triggers display of the menu and then draw the menu, and populate it with a handler or handlers for the items in the menu, so that a custom function runs for a particular item chosen


-] Q: what about shipping event handler together with view boxes?  And maybe shipping a menu function?  So individual boxes set the fill of the menu, while menu function does plumbing..  the Display expects a menu function that it can call to get menu contents..  Q: make key-based handling standard across different kinds of holders?  It seems src specific..
Maaaybe not..  any contents viewer will need to interact with the user..  the visualizer is specifically for human visual representation.  So a human will interact.  Sooooo, a menu system will be needed no matter what..  sooooo, the menu contents are provided by the holder, and rendered in menu format by the Display..


//==============================================================
//==
//==   Commander -- turning gesture into command
//==
//==============================================================


Example:
-] Previously a gesture caused the start of the mode for typing text into a field, say for entering the name of a variable
-] The gesture handler receives a new gesture event 
- -] the gesture handler looks up the current "mode", within state shareable among the event handlers.  
- -] The code then branches to where it extracts the value of the key pressed, from the event object.  
- -] When the mode started, a "collector" object was created that collects the key strokes -- when done, the user will give a gesture that causes the collection to be treated as a unit that is placed into a command that then goes to the modifier.  
- -] the collector object also has a link to the syntax graph object that is related to the text being typed in (cases: 1. an existing object is being modified, so that is the object linked to in the collector object; 2. a new element will be inserted, so the insertion point is linked to in the collector object; 3. ?).
- -] Idea: the related object is found as follows: 1. the event contains the Famous surface that the event happened on, get that.  2. the surface contains a back link to the view box it corresponds to, get that.  3. (one possibility) climb the view box tree until reach the view set, get that. (second possibility) the view box has a back link to the syntax-graph node it corresponds to, plus the name of the field within that node object 4. the view set back links to the syntax element, get that OR have the back link directly from the view box.  Now have the syntax element related to the gesture, OR the syntax node object related to the gesture.
- - -] Not exactly clear how linking from view boxes back to graph nodes works best..  might link individual view boxes back to, say, property nodes or port nodes if that proves to make things easier.  
- -] The gesture-handler extracts any needed info from the event object, such as which key was pressed, and adds that to the collector-object.
- -] The gesture-handler invokes the commander, passing along the gesture information and the syntax node related to it.

Thinking that things like moving edit-point (aka insertion point) and popping up the menu and navigation through the menus are all done by holder-specific event handlers.  The holder sends handlers to the Display, for example, packaged with the view boxes.  Those perform things that don't directly modify the syntax graph.  Only once enough gestures have been captured that a full sentence specifying a modification of the graph is ready, then does a command get packaged up and sent to the modifier.

So, the commander is the collection of event handlers that are attached to the view boxes!  

There will be common functions, so the view box handlers will be decorations that pass along to the core commander functionality.

Soooo..  how to decide which options are available in the menu?  They will depend upon the point in the graph where the edit point lies, or upon what has been selected, or upon which view box has received the event, and upon the current mode..  for example, can be laying down a connection between elements, or can be putting something into a port-box, or inserting a new element at a point in the graph.

Hmmm, thinking to make it easy, for now, all new elements are simply put down, with no connections, then wire it up.  Later, will add checks, that look at the types of ports and give a visual indication, direct from event handler, when two ports that are being wired are incompatible..  likewise, when displaying ports as boxes and subgraphs have an implied bounding box around them that is the output port of that subgraph, it won't let a subgraph be inserted into a port if the subgraph's output is incompatible.  These checks will all come from the holder, in the form of handlers given to the Display.  Might need to have some kind of round trip thing..  where event handlers send a command to the Modifier, which performs the check and sends a response back..  what's allowed will depend upon the details inside the graph..

But for now, just want the functionality of being able to build a graph!  Architect the code to be compatible with adding checks, probably inside the event handler, so that menus get populated according to the syntax graph details, and attempts to modify the graph are denied during the attempt, based upon details of the graph..  hmmm, guess that means need something inside the holder in addition to the modifier.. need an action checker, or referee, or rule judge, whatever call it..  kinda like referee, decides which actions are within the rules..  yeah, so make event handlers such that they can send attempted action to the referee, which is back in the holder, and get a response, then either allow the action or disallow with a message sent back by the referee.

Good.  I like that.. 

So here's where we're at:
-] Generic Display defined event handler
- -] It knows or figures out which view box (in a manner that depends upon the technology used to paint)
- -] from the view box gets the holder-specific event handler attached to the box.
- -] it calls the holder specific event handler in the view box
-] The view box's event handler is specific to the kind of visual thing -- a variable name, or an element shape, or a port box.
-] the view box's event handler figures out the action, based upon the type of event and the contents (IE, click vs drag vs keydown + particular key combo)
-] For certain kinds of actions, the view-box handler can make a call to the src holder's referee to see if the action is allowed
-] For menu actions, the view-box handler can make a call to the src holders referee to ask for valid contents to place into the menu
- -] the context of the event is sent to the referee or is available to the referee, such as which graph node is under the cursor, or which mode was previously chosen, or what nodes are currently selected.
-] The menu painting is implemented by the Display.  The contents and the handler triggered by each is supplied by the referee, and is relayed by the view box handler that receives the start-menu event, or relayed by the menu-handler that receives the select-this-in-menu (a keydown or a click) event.

Yes.  That covers populating the menu, based upon what is under the mouse and which key is pressed, and it covers checking the validity of attempted actions, and it covers advancing through the menu, and it covers making some parts be fixed things of the Display, while others are custom things of the src holder.  Yes, all covered  :-)

So, the outcome of the view box event handler is either a modification of context (such as zoom, pan, selected portions of graph, change edit mode), or it is a command indicating a change to the syntax graph (such as wire two ports together, or add new element, delete element or wiring, add property, and so on)
-] A command indicates a particular node or nodes in the syntax graph to be modified, and the modification to be made.  Or else indicates the addition of a new node or nodes, such as addition of a new element of a particular command type.  
- -] Most of the information in the command is taken from or generated by the code sent by the referee, which was the code of the menu handler, which was sent as the contents of a requested menu.  IE, user has a menu up, hits the key for a particular choice in the menu, that key is associated to a particular handler function that was sent by the referee, that handler function knows how to populate a command for creating a new Summation element.

-] The command is then sent to the Modifier, by the view box event handler or by the menu event handler. 
-] the modifier modifies the syntax graph objects that are indicated in the command.  
-] The modifier notifies the visualizer about the nodes that have changed.
-] The visualizer modifies the view sets of the affected elements, 
-] The visualizer then sends a notice to the display regarding the view set changes.
- -] Not clear on this part just yet..  does the visualizer send notice of which were deleted, which had contents changed, and which added?  Or does it just send the boundaries of a sub-graph that should be painted?  In order to send boundaries, those boundaries will have to be in-sync with pan and zoom, which are handled..  where?  Directly in the event handler, all inside the display?  Or, do pan and zoom make it back into the view set graph?  They modify the "do not cross" markers inside view set links!  Hence, the modifier must be the one to change those, hence pan and zoom generate commands.  Sooooo..  it seems just fine to have the visualizer just cause the entire current visualized sub-graph be redrawn.. it just tells the Display to redraw!  As simple as that.  It should, for example, be sure to block off any view sets that will be too small to be visualized, maybe adding a "too small" marker in the view-links, letting the Display choose an appropriate symbol, such as a small box or something..

-] Okay, looking like modifier updates the do-not-cross markers in the view-set graph and modifies which element is the root of the view.  The visualizer makes changes to the view boxes that correspond to the changed graph nodes.  Then the visualizer tells display to redraw the whole visible sub-graph, starting from the root!

Done.  That will all work.

Last thing is just calculating the curves that connect elements!  Haven't gotten to this yet..  in the graph form, have the arrows going between nodes..  in the Hamiltonian Path form, will have arrows that represent scopes triggering processor creation..  these arrows are not one-to-one with things in the view graphs!  Rather, they are implied by one node having a pointer to another node -- the existence of that pointer causes an arrow to be drawn from node that holds the pointer to the node pointed to.

Have to calculate the positions on the shapes where the arrow originates and where it ends.

have to add something to the view set that equates to the arrows, which has the data for the end points and the data for the control points that determine the arrow shape.

Let's see..  can have arrows inside a single element's view set..  in this graph view, each of the property boxes is visible and they are linked via arrows to the element node and the port nodes.

Hmmmm..  have to work this out next, as the next step in visualizing a graph..

Now, just need the stupid event handling to work like it's supposed to!
- - thinking abandon Famous.  Just post elements directly to the DOM.  Means have to calculate the shifts and scales myself..  see if can lift code from Famous for that..  All I need is the thing where build a tree of parents that have shift and scale in the link to children, and accumulate the shift + scale of ancestors.  Once the shift + scale chain has been applied, just add the result as a shape with the given size and position directly to the root of the DOM.

The problem may be that famous doesn't play well with SVG things, so have to attach the event handlers directly in the SVG string, which means have to post the generic Display handler in the HTML of the seed page, so that it can be named in the string that is inserted as "innerHTML" in the elements.  See whether can add a back pointer to an element and retrieve that from the event object, inside the generic handler..

//==============================================================
//==
//==   Visualizer -- generating view set graph
//==
//==============================================================

So, the way seeing view sets and view boxes and custom syntax shapes work is this:

-] in syntax graph, a property node states a visualization ID
-] that ID is looked up by the visualizer..  retrieves code that takes the graph element, crawls it looking for the visual properties it needs, then constructs the view set and view boxes and links those to the appropriate other view sets.
- -] The code for that ID can do anything it wants as the way it generates the view-set.  For starters, for, say, Summation, will have a fixed SVG shape that the code knows, and will have fixed locations around it for the input and output ports, and will have a fixed size for each of those ports.
- -] The input ports (and drawn output ports) each have their own bounding box.  Likewise, every view set has a total bounding box for the entire set.  The input port is part of the parent view set, so it has a translation and a scaling that is applied to the child view set.  The translation equals the position of the port (it is positioned relative to the same origin as the syntax shape).  The scale is calculated by taking the ration of the port width / contained-view-set-width  and the port-height / contain-view-set-height  and then choosing the smaller of the two.  When that is applied to the contained view set, then it will all fit within the port bounding box.
- -] Can insert view sets that are not attached to any element.  For example, an entire sub-graph, conisting of, say, 8 elements is inserted as the body of a summation.  Make a view set as the parent of that entire sub-graph.  The view sets of the elements are placed relative to this graph-wide view-set's origin.

Thusly, to make the visualizer code, only have to get an SVG string for the shape, doesn't matter what size it is!  It has a fixed size.  Then make the input port and drawn output port boxes of a size that looks reasonable compared to the shape.  And give them fixed positions relative to the shape's position.  There you go.  Done.  That now visualizes with appropriate sizes relative to all the rest, due to the thing where take ratio of bounding boxes as the scaling factor.


//==============================================================
//==
//==   Code discovery system -- reuse!
//==
//==============================================================
I'm sitting here writing all this front end stuff, and for sure someone else has written stuff very similar that I should be able to repurpose!  Yet web searches just don't catch it.  The current process for finding code out there that we can use doesn't work very well!  

-] People throw up their code, but don't talk about the patterns within it..  they put a vew paragraphs about the outside, using it as a whole, but that's it..  and no words about the interfaces, nor the modules inside, nor the mechanics/plumbing..
- -> A visual approach may help to quickly see the boundaries..  if the editor tool has a visual feature to highlight portions of code that are bound together due to names or shared variables, that will help to see boundaries quickly, so then can see the pieces..  then don't need words about the code, the picture shows quickly.  
- -> However..  still need to search for these internal pieces!  Improve this by allowing other people to annotate and markup the code base..  the system of pulling other code in by reference, and then adding mods over the top should help here..  when pull in code, can add local annotations, and the system sees the pieces that have been extracted..  the system also sees the naming, so when do a search, can just talk about what looking for, and the names use in search may match to names used in variables and functions and comments..  both original and those added over the top of references.

-] Going through someone else's code to figure out whether it will work for your purpose is painful.  Don't want to invest too much effort on something that will prove unusable.
- -> address by making it faster to evaluate other's code..  visual approach, features that show the stickyness between bits of code (for example, if find one thing that does part of what want, can enable a tool that shows all the code that gets pulled along with it, due to names or variables -- getting away from names as wiring should help this, and the explicit scopes talk about below may help as well)

-] boundaries within code are not clear..  what are the functional pieces?  The ways some people split their code up is beyond bizarre, and inverted ass backwards.  Discovering where the handles are that can grab and cleanly extract a piece takes a LOT of effort, have to learn large parts of the code.
- -> visual code may help here..  people see the shape of what they produce.. and can visually detect boundaries, given effective visualization (maybe 3D)

-] code is not cleanly separable.  Can't just grab a piece -- there are global vars and contextual vars that go with it, and it becomes like sticky spaghetti, grab just one strand, and it clings to another, then another, then whole big chunks have to be pulled out together!
- -> need language to encourage contained pieces that fit together..  explicit scope containers might help here..  no global variables!  Still are shared vars, but the code shows clearly variables that are shared, and shows what things get those variables.
- -> This will help make viable a tool that visualizes all the strands that have to be pulled in together
- -> Also, the type-driven execution (inversion of control stuff) sounds like it will help as well.  Although, this is a layer above basic POP, done via syntax extensions.  

= =] The challenge is to make POP gracefully support a variety of execution models -- just said a type driven model, want to allow ordering choices on the data.  Type driven has no ordering of datums relative to each other, any type that matches gets gobbled into any transform that accepts.  Want other orderings such as in-order (dataflow), or guard based (codetime), or something like annotations that state allowed pairings of datums (IE, door fits in particular frame, or a type of frame, separate the two, do things to them separately, then annotation says how to pair a particular door that has finished to a frame that is also finished).   Or even annotations that state order of functions (imperative style) or order of data flowing through transforms (stream based) or order of applying transforms to data (the data doesn't have to stay in order relative to other data, just the transforms have to be applied in a particular ancestry-of-data.. IE, track history of data, each transform is an edge, each node is a result of a particular path through transforms, has a particular pattern ((type)) for data.. so state something about allowed ancestry graph shapes).  Even control as data (compute some data which then acts as control for ordering of other computations.. is in INRIA notes somewhere, 2008 or so)
- -> Note that some of the execution models will support discovery and reuse better than others!  Allow the menagerie, but build the reuse system in whatever way works best for reuse!  It will inevitably end up biasing towards particular execution models, which will encourage code that gets written in that model, and encourage extensions on top of that(those) particular execution model(s).

= =] so, what is implied in basic POP approach, for supporting, say, imperative style, where the programmer explicitly states the scheduling of transforms on datums (IE, the application code tracks datums and commands particular transforms on particular datums)


- -> Seeing licensing being built in to system, as giving permission to view and reference..  anything with an open source license is automatically made available to the search system, which indexes across all POP instances, and caches so that if the original source instance goes down then the cached code is still available.  Code is identified by name, not by location within a particular instance.  The caching is peer to peer, so there is no central anything..  the indexes migrate around and update as circumstances permit.  May have name server infrastructure, so have fixed "locations" that a new POP instance will connect to, in order to tie into the POP system.  Those are the "bootstrap" servers, that act as entry points. 


========================================================================
=
=
=
========================================================================
Thinking about interfacing the javascript parts with the POP executable..

There's a Display piece and visualizer piece

There are Holders, which have visualizers..  and then there are processors, which may want to vizualize something..  and then there are name-spaces..  which are chosen to be defined as part of a processor.. 

So there's the case when have a running program..  which is a collection of live processors..

And have case when have a worksheet..  which is a visualization of a collection of live processors plus views generated by Holders..

And have case when seeing worm-holes out to other name-spaces (POP processor instances)..

And have case when viewing and browsing and searching..  will have other POP instances to interact with, they present visualizations that come through the worm-hole onto your Display.

Sooooo..  javascript is the Display implementation right now..  and it is the srcHolder, its Visualizer, Commander, and Modifier..  but the executable will be a C data-structure (graph) that builds itself out and a C based runtime that crawls the graph, generating inputs and firing off operations.

So, what's the interface between the C exe-graph and the javascript Display, and how do those relate to the CTOS persistent processors, the interconnection/enclosure of those processors, and the Visualization of the processors and their interconnection?  How does a Display relate to a worksheet and how does a worksheet relate to the mix of srcHolders and live processors underneath it that are visualized upon the worksheet surface?

-] So, I'm seeing for a worksheet that it Covers the entire Display, with perhaps a couple icons that can be expanded up in the corners..  So, can bring up views of available worksheets, and can bring up Service processors, such as for finding other POP instances that have relevant stuff to your interest, and so on..

-] I would say that POP instances advertise..  any that offer a worm-hole advertise what it will connected you to..  the equivalent of Google becomes a peer-to-peer system built in to the fabric..  when you do a search for things of interest, it is these advertisements are part of that..  and seeing POP instances that act as discovery clearing houses, maybe..  not sure how that's all going to evolve best..  but some sort of system of POP instances making stuff available via advertised worm-holes that can be discovered..  every pop instance has a Discovery Service processor, which in turn has available some inter-POP instance mechanism for going out and finding these advertised worm-holes..  I guess the two work together..  and the inter-pop service is a virtual thing that runs in all the machines that are part of POP instances, as a peer-to-peer mechanism.

-] In any case, have a worksheet -- that is a visualization of a collection of live processors and Holder visualizations..  so, thinking each live processor can have its own Visualizer..  
- -] Holder processors have Visualizers that paint the data inside them and have a Commander that provides a means for interactively exploring that data via a Display.  srcHolders add a Modifier for interactive editing of the data inside the Holder.
- -] other kinds of processors may also have their own Visualizer, which may connect to a Display.  Or, they may connect to their enclosing processor's Visualizer, or even the Visualizer of a worm-hole connected processor.
- -] There is a CTOS/POP standard for one processor to discover the existence of another processor, and then to request connection to its visualizer.
- -] There is also a CTOS/POP standard for discovering what kind of processor and asking about the Visualizer present within it.
- -] It is up to the individual processors to ensure that the requesting visualizer knows how to set datums that the receiving understands.  There is no standard for what a Visualizer has inside of it, neither commands (other than detection commands and the connect command and the accept command) 

In the case of games..  need very high speed access to the painting hardware..  so that comes in the form of a Display..  so thinking might have different kinds of Display Interface Language..  for example, make Direct X displays, and OpenGL displays, where the Visualizer talks to the Display directly via DirectX or OpenGL calls.  And then provide a wrapper around such Displays, so have the ViewGraph interface are defining right now for the srcHolder to make the MVDM loop.

(BTW, thinking about quality of stuff writing here, and envisioning big projects that may have been designed and were beautiful in many ways but died because they had too high a barrier between where are at the time of inception versus that end point..  obviously all this POP stuff and the physics-as-only-fields stuff with the computation model stuff are prime candidates for such a thing!  Seeing in inner eye a wasteland of half-finished art works, ambitious and beautiful, lying rusting and rotting away..  the end-goal was fantastic and the structure was worked out to arrive there, but the journey required too much change from adoptors, and too much effort to get to something usable..  chances are clearly high that this will all end up like that, the thought occurs to me..  Euoaaaa..  NO! Dammit.  Mmmmm, working for my pleasure, not really any particular end-point outcome in the world.  Just enjoying working out these ideas and giving them some form of life in code, as exercise while learning skills I can trade for money (javascript, Python, Raphael, requirejs, jquery, mongodb.. need to work in mongodb!) )

(BTW, what about writing the compiler in Python, called from the Bottle server, can start it now, as learning vehicle for Python..  and when have persistent processors, what about saving them into mongodb..  or better, saving all the source into mongo, to make it searchable..  start the discovery service that way..  add meta data about each source graph, just talking about it, and break that down via text mining, and turn it into an index for matching requests for code..  eventually add a recommendation system to track people's characteristics and match them with relevant code from others.   In fact, can add mongo now, finishing up that pymongo thing, and put in the )

=====================================
basically, the only thing inside javascript is the source graph and the srcHolder.  So the rest of the processors can be written..  what?  Inside C?  Have the Display interface.. hmmm..  what about making a javascript escape, and sending JSON through it..  could be sending to local host back and forth like I do with persisting the source graph.