Back | Home | Next

History

ANNA  K5

@

Home
Glossary
Objectives
Formalism
Morphology
Physiology
Connectors
Serialization
Traits
Methods
Claims
Relations
Dictionary
Core UML
Ring1 Apps
Tiger Server
Features
History
ToDo
Authors
API
Images
ToDo
DNA Declarations
Physiology RO
Real Time RO
ANNA as an Eco System RO
HMM Generator
ERP

 


The History of ANNA Kernel 5

The last entry is 344. Goto END


1 10 Jan 1999 Alexander_Mihail_INGRID First archive in PVCS.

One year of development on the PARSER project was not archived.


2 11 Jan 1999 05:04:18 Alexander_Mihail_INGRID

In this version all syntax elements are classes.

The class keyword is not needed.


3 14 Jan 1999 05:10:16

The Morphing Core can now load UML scripts and re-generate them to identity.

The matter of symbols used before declaring is still outstanding.


4 18 Jan 1999 03:16:42

Removed all unused functions. Removed SPI from TYPE.

Implemented TTree/TQList::Remove. A multitude of bug fixes.

Moved the responsibility of loading relations to TTree and simplified TYPE.

Added some error message boxes. Introduced TStream::Error.

Re-working the template mechanism lead to many algorithmic changes.

Introduced the File System-like TTree::FindCreate method.

Implemented the destructors and used them. Corrected them to relative satisfaction.

Implemented the UML loader as far as ignoring default field values.

More work on the TTree functions.

Problems are still observed on the chain of instances.

Another problem with forward declarations needs to be investigated.

It is difficult to delete between the heap and the data segment.

After all, the Morphing Core has set about removing that discrimination.

We cannot go to the next step, of morphing with default values, without an engine with all these algorithms running perfectly.


5 22 Jan 1999 05:32:44

Corrected class name parsing between the loaders of TObject and RELATION.

Names are now allowed to have spaces only when enclosed in angle brakets.

Resolved the recursion problem when the failing reduction of a RELATION has to remove the forward declaration (fh) that it just introduced.

The symbols table is now correctly built. No errant, extraneous symbols.

This is the first decent compiler output.

Difficult work was necessary to correct the destructors.

The Initialize methods were needed by class masters such as TMethod who contain other classes by value.

This is to properely bind the containd components of an instance to their associated Type Masters.

Destroying the static Type Masters is a delicate recursive process, so far non deterministic in the context of C++ constructor/destructor serialisation.

We therefore manually delete the categories that were loaded from the input file, and instruct the embedded type masters to not recurse with the delete operator. The overloaded delete operator and DoDelete flag govern that choice.

Corrections to various string comparisons that were not expecting anonimous items.

Removed the unnecessary method formats.

There was valid reason for TTree::Insert to first remove the item from its original tree. That allows items to be swapped between two trees. The condition

arose in the handling of forward declarations.

The TTree::FindCreate method was exercised and corrected.

The order of those abstract Type Masters is crucial for the compiler.

The order of syntactical reduction should be TMethod/RELATION/TYPE/fail

TYPE re-aligns all its clients upon destruction. It also destroys all its instances. That triggers a recursion through the destructors.

The loader for a TYPE has changed to handle the recursion that arose at class declaration when contained fields reference the incomplete type.

All other clients of the forward declaration are assimilated by the newly declared type, and the forward item is deleted.

The loader for RELATION was generating rubbish forward declarations whenever it failed. It never actually directly failed, but that was another flaw. Due

to suspicion of another bug, the rubbish is still left behind, but provisions are made about it. See the fh (forward handle) variable.

Due to re-entry, all destructors must initialize the class fields.

The known outstanding problems are: indirect scopes, forward declaration destruction, recursion through destructors.

Cleaned the sources-up to be ANSI and built under MSVC v5.

Built an ALPHA target that successfully ran on the SIX NT server.

Names are now allowed to have spaces only when enclosed in angle brackets.

Class modifiers.

TRoseValue uses the Information field to store loaded values.

Values are generally malloc-ed but booleans are directly stored and should never be freed.

The RoseSuperClass was removed as non essential.

The Initialize methods were needed by class masters such as TMethod who contain other classes by value.

This is to properely bind the contained components of an instance to their associated Type Masters.


6 23 Jan 1999 23:44:50

Destroying the loaded Rose hierarchy to prove the destructors.

At phase 4, I am now showing CategoryAll before and after loading the script.

Not all relations were bound to their Type Masters due to construction order.

No more anonimous keywords in the generated scripts.

To give templates to the method prototypes or implementations was wrong.

Corrected all of the above problems.


7 24 Jan 1999 05:22:44

Removed the call to create a template when compacting the origins.

Compacting origins is some trick we have to do to get around constructor problems.

Create cannot be called for types other than TObject descendents.

The stream loader can only invoke Type Masters that are TObject

descendents, as only those may have Load methods.

With those clauses in mind, some logic was simplified.


8 24 Jan 1999 11:12:58

A subtle recursion of the concepts occurs with TYPE instances and descendents.

Classes loaded from scripts are Type Masters in an internal sense but in the scope of the script they must be initially empty and then different types.

This dual identity is the first orthogonal recursion shift.

The core algorithms must be applied in the form of normal function calls over the symbolically assembled structures.

The core mechanisms depend on the ability to precisely establish the identity of a type.

I've relaxed the tests so that two types can be compared symbolically.

The only algorithm that can work over equivalent types is the instance creator of TYPE.

This step is crucial in implementing default field values, and soon after morphing of instances as the compiler completes the types.

Scripts with templates and field defaults were generated.

This is delicate subject matter.


9 26 Jan 1999 10:55:06

Size and offsets managed by the core are now bit granular.

Some char pointers have become new lstring/zstring fundamental types.

Bool, point, float are fundamental types now.

The template binding algorithm should operate symbolically. As it doesn't, the order of fields in TObject needed to change to keep the used pointers at the top, and externally defined lstrings at the bottom.

Some stubs were added expecting instance resizing and type morphing.

The stubs for rendering dialogues were removed and will be reintroduced later.

The VMT is modeled with Rose, but the field is not generated. That's to not corrupt the scripts.

The compiler looks for an empty bool type declaration to give it a size of 1.

All other types can be built from that.


10 27 Jan 1999 19:32:36

TYPE uses StoreInstances for symmetry reasons.

This version allocates/clears/copies memory at bit granularity.

Generating short declarations on one line for more compact output.


11 28 Jan 1999 02:29:50

Corrections made for MSVC.

Showing the type when generating an instance so that the output can be recompiled. Instances are only generated as a debug tool.

TTree::Insert was turned around.

Insert/Remove are now consistently virtual so that specialized types can take special action.

RELATION sets its offset field when insertes in a type master. This is a critical step in acheiving instance morphism.

No need for TYPE::Insert/RemoveRelation.


12 31 Jan 1999 01:58:50

Class instances have a pointer back to their containers.

When creating a class instance MaxSize of memory is allocated anticipating morphisms of the Type Master.

TStream was extended with MatchPair to skip past bracketed expressions such as initializer lists.

The core loaders were changed to carry-out their semantic actions after syntactical acceptance.

This is to reduce the strain on the symbol table.


13 05 Feb 1999 05:52:50

The output of the compiler was split into debug files and main target file.

Implemented RELATION::LoadFieldValues.

Although this version of the Morphism Core runs well, it highlights the true dimensions of the problem domain.

Morphing types and instances is further away than ever. A VMM with clustering needs to be implemented.

Double indirect pointers need to always be used. The loaders need to be pre-conditioned with the symbol tree.

Much more thought needs to go into the handling of the Forward Declaration Stack.

Relations need to have a size and therefore be "windows" of their Supplier type.

Building an instance should invoke a "Paint" method.

Out-of-bounds fields need to be clipped or cached for readable fields.

Relations need to be syntactically extended to be R/W.

Implemented RELATION::LoadFieldValues.

My own atoi function that can interpret hex values.

The template mechanism has changed yet again. Templates are now passive, unlike instances. Nobody other than their Type Masters point to them. The template is not an instance of its type and doesn't chain to Instances.

To resolve forward declarations, a Type Master allocates a template larger than the expected type's size.

TYPE destroys its template explicitly.

The loader for TYPE has changed to introduce its template or reduce the forward declaration stack before loading its relations.

The syntax for field initialisers has been unified with that of types: braces and semicolumns. To resolve a syntax determinism problem, initializers are introduced with the "=" sign. The syntax remains unsettled.

To allow forward declarations, the initializer list must be deterministic and the size of the type must be infered from the list of initial values. Empty default initial values lists are no longer allowed, and the mechanism of reducing current values against templates is not used anymore.

This version can compile without needing to merge with the core types.

The relative inter-path algorithm was disabled so names must be unique.

Changes to TQList::Insert: we are allowing insertion in null targets to make the TTree tests on Children simpler.

TYPE will need to become the Virtual Memory Manager.


14 10 Feb 1999 20:48:46

This version works very well.

Provisions need to be made for an internal class comment.

Simplified the string of symbols used in Load/Store. More work is required.

Corrected the comment loader to not eat newlines.

Hierarchies can have a leading comment now, which is kept persistent through Load/Store as the Version string.

The loader of method prototype/implementation hierarchies also creates a template so that instructions and default function parameters can be coded in.

The syntax of the initializer lists has reverted back to before Rev 1.15

Anonimous has-relationships are again allowed. Paranthesis and commas are used as separators.

One set of parantesis is enough around complex structures.

The equal sign is not used.

The loader of initial values doesn't take into account the TObject aspects of the fields, and it has been cleand-up in subtle ways.

All output is correctly fixed-up out bit boundaries and the first instructions were coded and were preserved by the compiler.

Forward declarations were handled correctly.

TTree::Find was added and TYPE::Find was changed to use it.

Propagation of bit granular offsets is more homogenous now.

RELATION will soon handle the Size field, to form a window over the supplier.

Loaders and storers of all kinds are now passed the types and forward

symbol tables. Corrections to the bit-stream manipulators.

The size of the templates is now configurable and set to 0x1000 by default

and can be reset externally.

Generating the dbg output after the target script.

Simplified the output of the compiler to not contain any internal category

and only what was loaded from the input stream.

Numeric constants were added to the language.

TConstant is new and handles the loading, storing and matching of numeric constants.

Enumerated types can now be built with classes containing constants as fields.

Enumerator types can descend from one another.

An instance of a hierarchy of constants has the size of the largest constant in anticipation of an assignment.

As a type might consist only of constants, its size will be 0. An instance of such a type needs to query the GetMaxConstant method to fint its size.

There are no more fundamental types in the language. All numeric types are built out of constants that define their numeric boundaries.

I am beginning to understand numbers better. There are many issues left unaddressed such as values that are not mapped by any constants.

TTree::Find was added and TYPE::Find was changed to use it.

Propagation of bit granular offsets is more homogenous now.

RELATION will soon handle the Size field, to form a window over the supplier.

Loaders and storers of all kinds are now passed the types and forward symbol tables.


15 12 Feb 1999 02:05:16

Original test script as loaded and generated by the Morphing Core version 1.20.

This version of the compiler handles pseudo-assembler statements and function calls that involve passing constants, local and global symbols as parameters.

The outstanding matter is calling functions that were not declared yet and loading scripts with reduced initializer lists.

The Before function was added to help compare two tree nodes in terms of their

relative position in the tree. It returns true if l can be found before r.

This function is tightly coupled with the direction of the Find algorithms.

Since Load methods receive zeroed instances, clearing of fields is not needed.

TTree::Find is now virtual and overriden by the descendents. It takes a type master as a filter and a direction.

One can attempt to find an object down the tree, or up amongst sibblings and parrents. TTree::Root is back.

TYPE overrides the Find method to do a flat search on the first degree children on an upward search. In other words, a type exposes its relations.

TMethod overrides the Find method to return the Prototype pointer on a match.

RELATION::Load doesn't call FindCreate anymore. FindCreate is orphan.

Store methods can attempt to reduce the actual data against defaults. That logic was improuved and enabled/disabled with ReduceFieldInitializers.

Since the name resolution logic now works bottom-up, the Forward Declaration Stack tree, gen, cannot be inserted in CategoryAll until it's compiled because

loaded scripts use the same names as the internal symbols.

Previously, the Forard Declaration Stack was searched top-to-bottom.

That was wrong since unwanted tree nodes from unrelated scopes that were declared before could respond to Find queries.

Inverting the scope resolution logic to bottom-up involved first chaining the tree elements and than carrying-out the semantic evaluation of the compiler. That was a delicate process.

The templates were simplified and they do not need to be set-up by Create.

The syntax of the language was extended so that any name of a relationship starting with @ can come before the rest of the syntax.

This "name-first" style of writing is needed to emulate labels in assembler statements.

Over all, the logic was cleaned-up and areas of ambiguity were clarified.

Added the rev info string.

This version of the compiler handles pseudo-assembler statements and function calls that involve passing constants, local and global symbols as parameters.

The outstanding matter is calling functions that were not declared yet and loading scripts with reduced initializer lists.


16 21 Feb 1999 17:45:32

The semantics of the language have changed. Functions and relations are now classes.

A class can have return types like functions do. A class, in other words is a rule of recombination from the class fields to the return type's fields.

Many mechanisms were simplified and generalised as a result. RELATION and TMethod are gone and all work is done in TTree and TYPE.

The problem of calling functions in advance of their declaration was resolved.

Further refining is neccessary to allow symbols to be resolved from the ancestry of a scope.

The Prev field of a TQList is now used and correctly chained.

The internal symbol information was reduced to a strict minimum as updating it hindered the development.

All constructors were changed to pass all field values.

The internal symbol information was reduced to a strict minimum as updating it hindered the development.

All constructors were changed to pass all field values.

TTree::Find was extended with the flags parameter and the Match(flags) callback.

The callback can be overriden to use the flags and discriminate between Relations, Methods, Classes, etc. which now identify themselves with booleans in TYPE.

TStream was extended to allow nesting of files.

CompactOrigins was removed as it became obsolete.

LoadComment is a better version of LoadInformation.

The difference between class fields or function parameters and implementation

fields remains outstanding and will be dealt with in the more general context

of static data definitions. A static member is one that has visibility in one

scope but allocation in another.

The Rose loader was developed further to read distributed models. It cannot expand Virtual Path macros yet.

Some factorisation of code between Phase3 and Phase4.

The new loader can understand v42 models.

Default constructors were removed as better template setup options were found.

The two-word types were removed as they are not supported by UML.


17 27 Feb 1999 17:00:14

This version addressed the matter of instantiating the prototype of functions with a block of instructions.

The implementation section of a class stores default values into a second template. The Template and Size fields of a TYPE are now vectors of 2.

The Phase count is changed between the public and implementation group of members. The phase 0 of a Hierarchy has the size of all non-implementation data fields of that chass. This is how functions can be called ahead of their definition. The consumption of template size has doubled.

Morphism must be implemented.


18 03 Apr 1999 04:48:32

The language was extended with for/if/switch/goto statements which work fairly well. Work has began on the grammar/compiler for expressions.

The chain of loaders has changed.

Work has began on the grammar/compiler for expressions. EXPR.CPP is new to the project.

The language was extended with for/if/switch/goto statements which work fairly well.

Work has began on the grammar/compiler for expressions.


19 01 Apr 1999 23:08:08

More separators. Separators need to be served from the outer layers.

Changes to the symbol find logic.

TTree::ParentStore is gone.

Load/StoreChildren is now virtual. Load/StoreChild are new virtuals for specialized types to override them and take additional action such as symbols fix-up or reject when called from LoadChildren.

TLabel is new to the core and formalizes the public/protected/public matter as well as ordinary code and data labels.


20 26 Apr 1999 17:41:04

Fixed the loader of "return" statements who suffered from the flaws in TRemGrammar::Load logic.

Some clean-up and some fixes for ALPHA.

A compiler parameter to enable stream debugging.

The core was enhanced with grammar based loaders.

TGrammar is a tree that points to a TYPE to be used in creating instances.

The Load(TStream*,TGrammar*,TYPE*) is then invoked to load the instance or construct, load and return an object of a different type as determined by the grammar and the input stream.

Sibllings of a grammar instance inplement the alternatives of a production.

The children of a grammar instance establish order of precedence. It's a complex matter.

TRepGrammar specializes in loading priority lists such as arithmetic expressions.

TStream was enhanced with a Get(TGrammar*,TYPE*) method.

TExpression needs to belong to the abstract core to implement the language independent cross-refferenceing system.

The compiler can now interpret arithmetic, assignment and primary expressions.

Priorities of operators and paranthesis are now handled correctly.

Production variants are demonstrated by the statements logic for conditions that all use the GExpression multiple production.

The hierarchy of statements has changed signifficantly.


21 28 Apr 1999 23:35:06

Added the grammar for unary and conditional expressions. Better hierarchy balance for expressions.

TRepGrammar was removed.


22 03 May 1999 20:36:24

Implemented the destructors. Overall code reduction.

Added the grammars for Postfix Expressions and function calls.

The syntax of C++ is complete minus the casts.

"." is now a separator.

Flags are used in TTree::StoreChildren to select braces or paranthesis.


23 13 May 1999 19:31:36

All grammars are now statics of their associated types.

Member selector expressions keep a string with the member name awaiting the enhancment of the forward declaration logic.

Better code to bind expressions to the symbols table.

Enhanced ForLoop loader to interpret do and while statements. Support for do and while statements.

The flags used by the Find routine got confusing and needed to be symbolic.


24 A 1 year gap

{

Check the activity logs. Did I work on SATURN?

}


25 Friday, May 05, 2000

The parser's main was renamed to main_parser to avoid clashes with other modules.

Moved from PVCS to SorceSafe. Port to BC5.02.


26 Tuesday, June 13, 2000

PROCESS was ported to BC5 and included into the OS image.

Removed Void, Int and Char from the embedded type masters. Split the helper functions out of here iand into AbstHelp.CPP so that the remaining Abstract.CPP can be assembled in limited RAM space.

The COMPILE category was invoked to create an instance for the script loading process and failed to see that Size[0] is 0. Corrected.

Resolved some of the problems of calling the constructors of static data.

Sources were regenerated from the Rose model. Added components to Arcade Toolbox. Added stubs to satisfy the linker for MPEG, TrueType and Arcade.

Corrected the virtual path of the model files for use in Saturn OS. Removed the CR\LF from the model comments.

Included the Daemon to the project. Got the Daemon to work with the OS.DLL

Reconnected the Rose model.

Our printf cannot cope with null %s-es.


27 Sunday, June 18, 2000

UNTESTED cannot be empty and it axpands to ;

Process was moved inside the Morphism Core. All dependency on TREE was removed and replaced with TTree. No MSVC code.

TTree::Find shold only recures for upward calls. Added TTree::AddChildren for the Enterprise category.


28 Thursday, June 21, 2000

Two important sets of changes were needed to get ANNA's reose model loader working:

1. TypeVoid, TypeChar and TypeInt needed to be re-introduced for the relations created by the artifficial inteligence in the rose model loader so that the offsets into the type instances are correct.

2. Due to changes to the forward symbols resolution mechanism in ANNA's Morphism Core, we need to relax the failure on relationary reads/writes to inexistent fields of class instances. We cannot assume that a field exists.


29 Sunday, July 02, 2000

Labeled 'I've seen Glide working for NT.'

Ignoring the case of nested Petal files.

Changes to the Memory Manager.


30 Friday, October 20, 2000

The functionality of TYPE was split in two classes. TBaseHierarchy controls matters such as CreateInstance, Size, Instances, IsType.

The Remaining TYPE deals with Relations and script compilation/generation.

The Base Hierarchy was needed to export the Enterprise in a minimum form, yet able to Serialize.

Hierarchy was split away from Abstract. Added the Serialize functions. Reduced a few Enterprise fields.

Extended the Hierarchy Master with double indirect virtual table logic.

Added the Escape character to the blanks.

INIT/DONE replaces C++ the C++ constructor/destructor. Added the CAT containers. Got rid of the Templates. Implemented the Host file binder for TStream.

TStream is not abstract any more, and it implements the Windows file IO. DisassemblerStream is now redundant. Certain CATs need to register their Pivot pointer for other modules to use.

The CAT initializers retrieve the VMT pointers and record them into the coresponding TYPE::VMT_NORMAL field. Added the default constructor.


31 Monday, October 30, 2000

GUI: Original CBuilder application to view an Enterprise.

Building the Enterprise as a DLL.

Moved the initialization of the Enterprise Type Masters out of the tester and into the InitBHrc module.

Simplified the Test structures. Made the tester better. Marshalling a string of commands that are then switched in a while loop.

No more static counters of Enterprise instances. Using the InstanceCounts of the associated Type Masters.

Changed the constructors of the Enterprise to INIT methods. Corrected the constructors/destructors of Enterprise objects.

Corrected the GuiClients logic of the Enterprise.

Added the Interlocking structure to LINEs and THREADs.

Added Set/UnsetTypeMaster functions to LINEs and THREADs to deal with the Interlocking structure.

The Null node/line/thread are now constructed only if BindTemplate is called.

Added constructor/destructor for NODE_CLIENT and NODE_VIEWER.

Reduced the TYPE::CreateInstance code with TBaseHierarchy.

Added the TBaseHierarchy::FindByClassID function that is used by the pointer serializor to make the binary files relative to Type Master IDs.

Moved all Type Master interlocking logic into the Set/UnsetTypeMaster virtual methods to allow descendents to override it. This is to do with contained instances.

Corrected BaseHierarchy::DONE to consider contained objects when deleting the instances. It now calls the instance container's destructor. Maybe it should change to just detach the instance.

Added the TBaseHierarchy::GetPivot static function as we don't seem able to export pointers such as Pivot.

Using the MORPH namespace.

Made the Tree::Store backtracking mechanism to use an externally configurable MaxLineSize variable.

Changed TObject::INIT to strduplicate the names, to zero the serializer cache and the container pointer.

Moved all Type Master interlocking logic into the Set/UnsetTypeMaster virtual methods to allow descendents to override it.

Corrected the TQList::DONE so that it first removes the element from the list.

Corrected TTree::DONE to not delete a child node if that is contained by another object, and remove the child instead.

Changed most INIT methods to assume a default type when none is specified.

A multitude of changes/corrections to the pointer serializer to deal with

contained objects.

Added UIDS.H. Using Class IDs.


32 Monday, November 06, 2000

GUI {

Added the tests for Saturn.

More complex test structures with endless loops of wiring.

Using the NODE::Expand function. Implementing the TIsNodeClient::InsertItem virtual function do the Enterprise controlled expansion of the tree viewer. New logic to trigger the GetSetFormat methods of the BaseHierarchy on double-clicking the Type Master grid.

Another zero-virtual function to get the face of the Hierarchy.

This version was used to view Enterprise structures of the whole Saturn build.

Moved the construct and delete to the local menus of both tree biewers.A derived stream to route WriteBuf to a TEdit control.

Local menus for the tree viewers. Much better logic to manage bitmaps for ANNA's Type Masters.

}

TBaseHierarchy descends from LINE instead of NODE (for now).

Using get_Supplier and set Supplier instead of Supplier directly. That's to get closer to using the From and To relations of the LINEs of the Enterprise.

GetFace, GetSetFormat and RenderFace are new and delicate methods to allow calling some virtual functions without an object instance. A bug fix destoying the hierarchy objects.

No need for the NODE_VIEWER class after rework of Expand/Collapse methods and the introduction of NODE_CLIENT::InsertItem method.

Node bounds and positions are all called attributes and handled by the new GetSetAttributes method.

Got rid of sys/stat.h.

This CLASSDECL dependes on the "CORE" define and expands to either export or import. Using #defines instead of enums so that the CLSID_s may be used in the resource files.

TBaseHierarchy::Pivot. Get/Set Bounds/Position.

Fixes to Pivots, Fixes to the indent=depth1-depth2 calculation. Initializing the instance data to zero by using calloc instead of malloc.

Changes to reflect the new namespace MORPH.

Corrections to FindByClassID and DescendsFrom. New logic to deal with bitmaps in GetFace calls.

Resolved some of many problems with exporting/importing Pivots.

Get/Set Bounds/Position functions.

The Interlocking structures were initialized twice and kept incrementing the Instance Count.

Added resources for Type Masters responding to GetFace calls. Added ABSTRACT.RC


33 Wednesday, November 08, 2000 Labeled 'TYPE==RELATION'

GUI {Changed the path of script files.}

The last version with unified TYPE and RELATION. This arrangement proved to be prohibitive. I'm still vascelating.

Reduced the Pair of LINE end nodes to TTree instances.

Removed the HiddenSupplier and discovered a destruction problem when using the NODE::get_To() function instead.

Removed Clients and NextClient from TYPE and broke the TYPE type loader. This loader needs to be grammar based so that it may delete the "this"

instance an replace it with the forward declaration found.

Removed the boolean relation specifiers and replaced them with a Flags field.

The constructors are now simpler. Thre are too many paranthesis.

Templates are back to support the older style of morph assembler scripts.

CreateInstance may be invoked by a type with a zero VMT (a non-object).


34 Monday, November 13, 2000

Two major changes have taken place:

1. TYPE and RELATION are now different classes. This is how the Morphing Core begun anyway. I was not convinced that types and relations can be one and the same concept. I am still uncertain either way, but right now, relations are many and to handle them as type masters seems prohibitive. RELATION is a LINE descendent and TYPE is a NODE.

2. The script loader was consolidated to be grammar based. Grammars are now formalised better as trees constructed differently, with a null Prev (this is how we distinguish grammars from trees) and using the Parent pointer as the Type Master Loader. Next is the "or" clause, Children is the "and" clause. Things are nicer and clearer now.

3. Methods have become a relation between a scope and a return type.

The Rose model loader demonstrates basic morphism, THierarchies and RELATIONs, as well as the new grammar based loaders. It works.

The Store path needs rework since it was node oriented and now relations are lines and they do not appear in the listing. The native script compiler does the

syntax analysis well but is not binding the symbols table properely. I need to think more about the meaning of expressions as lines connecting the types involved.

All in one, this has been a difficult transformation of the Morphism Core.


35 Tuesday, December 12, 2000

InitTypeMasters is provided by the InitBaseHierarchy module that replaces TYPE for non-ANNA applications. This is a stub that returns the master pivot.

NODE overrides the Insert method to destroy any GUI handles that the node may have and to notify the GUI clients of the target node that a new child was inserted. NODE_CLIENT::Insert is a new virtual method to notify a user interface that new children were inserted in the specified node. Added two new topology Experts: NODE::IsPath and LINE::IsOnPath. NODE::IsPath

returns the first line in or out of a node that would lead to a given destination. LINE::IsOnPath tells whether the line lies on a path between two given nodes.

Corrected the THREAD::Serialize function.Added the THREAD::Evaluate function which invokes the evaluation process recursively on any line that ends into the thread.

A bug was found and fixed to do with deleting an instance of a Type that is its own Type Master. Thiese instances are the generators of the Morphing Core's types.

Corrected the set_Name and set_Information functions.

Correction to TYPE::Load to not erase the name of a function prototype which is loaded by RELATION. Relations are not inserted as children of the Hierarchy.

Added the pick correlator for LINE. Modified the Node correlator.

Load/Store methods for the MORPH category were updated in the Rose model to reflect the new grammar-based loading scheme.

Corrections to NODE and LINE destructors/constructors. Implemented the NODE::Load method. Re-implemented the TYPE::Load and RELATION::Load methods. These work on a simplified ANNA3.CPP assembler script. Corrections of the Store channel. Corrected the grammar of the for statement. The following problems are still outstanding:

1. This version of the compiler produces a tree of relations and no hierarchies.

2. Statements are stored twice.

3. The IsBad morphism didn't work.

Heavy changes to the compiler and to the language grammar to have TESTSCR2.CPP loading again. Binding the symbols table is still too primitive to be real.

GUI {

Linking with ANNA K5.

This version of the viewer has loaded the full Rose model of ANNA K5 and displayed it. Circular relationships.

Showing the names of the relations. This version of the color palette doesn't crash.

}


36 Monday, December 18, 2000

Enterprise objects now go to tmp32\entprise

Use TObject::Free to avoid scope confusion with CBuilder.

GUI {

Changed Display Layout for free-form.

Added trace windows for Tapes, Lines, FastLines, Points, Area, Bars, Pies, etc.

Added Animator, Picture, Comment and Log windows. Added FastChart windows.

Changed the #ifndef IS for #ifdef OS to allow both. Changed the DLL dependency. Lost the OS.LIB.

Added the IS Evaluator logic.

Added the Checbox ICam. TICheckbox and TIDisplay are both TIHost ICams.

Removed the Attributes for nodes.

Introduced the TRuntime class. The Runtime class has virtual functions to create a checkbox and a trace. A recursive Enterprise traversal invokes the nodes to attach to the runtime class. The GenericInput ICam has a simple function parser that recognizes sine, random, square and ramp functions with some parameters.

The evaluator for arithmetic ICams is looking for names of "+", "-", "*" and "/".

void TIsNodeClient::Insert(NODE*node)

Changes to the Type Master reporter and editor. They're on the same tab now.

A better way to initialize the toolbar of Type Masters.

Deleting the current node is better and it resets several variables that hold the current state.

Drag and drop logic for the Scope Viewer.

Editing the names of tree nodes now refreshes all known viewers to reflect the change.

Heavy changes and enhancements to the drawing algorithm. Resolving loopbacks and highlighting the referential closure of a node.

Changing the selected node now updates the editor's toolbar to reflect the new state.

The Schematic Editor allows new wires to be set by double clicking onto source and destination nodes.

Changed the Parameter Sheet logic. Functional Accept and Reload buttons.

Added the handlers for Instrument Run and Pause.

Added some experimental thread logic to run the evaluator and refresh the charts.

Several other changes.

Added checkboxes and trace displays.

Added the GUI handler for the MORPH assembler scripts.

Changes to support the LINE correlator.

Enabled the LINE pick correlation algorithm. Thick selected line. Line name on the joint.

}


37 What happened here???


38 Tuesday, March 27, 2001

GUI {

Wide screen aspect for Carmen.

Got to revisit deletion of Enterprise nodes.

}


39 Wednesday, April 04, 2001

The constructor for RELATION should not allocate memory. Reworked the To and From initialization.

Added new. Fixes to construction problems. Modelled the Free function from Rose.

Corrections to BindTemplate, BindThread and RunScript that have fallen behind.

Initializing the expression TypeMaster.


40 Sunday, April 15, 2001

Ported the Enterprise micro kernel to Visual C++. The problem of acquiring pointers to vftables is still unresolved.

Changed the logic of tree insertion so that Children is always the last element inserted.


41 Thursday, April 26, 2001

GUI {

Simplifieed the panel and splitter arrangement by docking the class browser and instance editor. Added the Filter tab. Changed the position of the

TypeMaster icon bar. Rearranged the test menus.

Paths to test scripts are now the same as for SATURN.

chdir("..") is to get out of BIN32 to the same level as ALL.IDE so that the tests are spawned the same for SATURN and EDITOR.

}

TStream was changed to descend from NODE instead of TTree. This is to establish the base of file locking scheme with LINE semaphores.

TStream needed to move into its own header and CPP file split away from ABSTRACT. The sources of the morphing core were re-generated to reflect the new inclusion system.

Removed any assembler by using the new GetVMT function that constructs a default class instance just so to return the VMT pointer.

The new scheme is needed for porting to MSDEV.

Enhanced the Serialize function to pass the version count through.

Changed the Pivot_Object pointers back to TObject::Type for a consistent way to get to remarkable Type Masters.

Setting various shape flags for NODE descendents.

LoadMdl and ProcTest testers were turned into class::main(argc,argv) static functions. This is a new convention through-out ANNA K5 for testers.

Added stream.cpp. Deleted PROCTEST.CPP. Added matrix.cpp.

Implemented the ENTER macro.


42 Saturday, April 28, 2001

Changed the Pivot strategy back to ::Type statics.

Changed the logic to acquire VMT pointers away from platform-dependent assembler and to GetVMT functions that construct transient instances to get the VMT.


43 Friday, May 04, 2001

Converted the Rose Model of ANNA K5 from V35 to V40. Re-generated and corrected the source pool.

Renamed TObject to OBJECT, TTree to TREE, TStream to STREAM and TLabel to LABEL to avoid having to use namespaces. More work on the TFile file system.

Changes to OBJECT::Free, DONE and destructor.


44 Thursday, May 03, 2001

GUI {

Renamed TObject to OBJECT, TTree to TREE, TStream to STREAM and TLabel to LABEL to avoid having to use namespaces.

More work on the TFile file system. Changes to OBJECT::Free, DONE and destructor. Changed the point definition.

Added TMatrix and rect. All sizes and offsets in the Morphism core are now (x,y) points and the Type Master initializers needed to change accordingly.

}


45 Thursday, May 17, 2001

Initialized TFile->TypeFlags and TFile->Position. The Position needs rethink.

Implemented TFile::Seek to remember the current position. Currently it clips between 0 and GetSize() values. That may need to change to adjust the size. Enhanced the TFile::collect_detail to include the filename in the buffer.

Defined TFile::Serialize. Changed the UserData from void* to OBJECT* and implemented TFile::Serialize to route through the UserData pointer.

Isolated the constructor of CAT_FSYS that only makes sense during builds with the TYPE of K5's Morphing core.


46 Tuesday, May 22, 2001

GUI {

Added the Instance browser and the data formator controls instead of the

Parameters tab. The navigator cannot be kept unaware of the Morphing Core

anymore. Added a set of flags to configure the drawing effort. Removed the Rose

model loader tests and enhanced the FileOpen logic to load any model file.

Back to mixed case.

Simplified the Instrument Server.

Renamed ISCTRLPN.CPP to EntNavig.CPP

}

Corrected TYPE::GetFirsRELATION.

An attempt to correct the Universal Assembler Script loaders. More work is required after heavy changes to the Morphing Core.

Protected all boundary iterators so that only LINE descendents are considered valid for containers of TREE endpoints. LINE::GetNext didn't work properely resulting in less visibility.

Added an enum with speciffic codes for LINE visuals: uses, has, inherits.

Cleaned-up an unused symbol called $UNNAMED$ or something. Layed-out nicely.

Re-implemented the autoinsert scheme at INIT time. Only NODES and their descendents may use it since the TREE class can also be used as a GRAMMAR and parents and children then have a different meaning.

Exporting class RELATION for the Enterprise Navigator to import.

Implemented the STREAM::Seek function to go to the host O/S.

Don't want to include HIERARCH.H in client applications.

Added bitmaps for the FSYS clategory.

Added the OBJECT::DebugResolution field from ANNA K4.

Added DEV.MPP

Got the File System test working. It has bugs and its design should be superseded by the Morphing Core.

Changes to DONE and Free. Isolating the new and delete operators for MSDEV builds.

Corrected the initialization of the FSYS category. It used to shunt the Hierarchy instance chain.


47 Saturday, June 09, 2001

Merged the Load and Store methods with the new Serialize method created in the previous revision.

Reduced STREAM::Get in favour of the new grammar-based SerializePObj method.

Got the Hierarchy and Relation storing to work over text streams. Got the Rose model loader working again with the new Serializer algorithm.

Consolidation of serializor along the TYPE for reuse of both code and sequence.

Enhanced the STREAM class with a virtual method to set the indent level for serializing text. The GUI overrides this method to drive the indent level of the Rich Edit control.

Moved some virtual methods of CANVAS to STREAM level. All streams may now set fonts and colours.

Changes to the GUI ScopeBrowser to load the CurrentNode into the RichEdit control via the new Serialize method.

A problem still remains with the clearing of the SerializerCache at the NODE level when mixing Paint and Text output requests.

All unspecified nonclasses need a comment to explain their size in bits. The newest Rose Model loader interprets the documentation field of nonclasses and sets MaxSize.x with it. These are needed to resolve GetSize calls.

CAT_MORPH, CAT_ABSTRACT, CAT_INTERNAL are type pool generator and instances of themselves. Enhanced these classes and class masters with GetVMT caps.

Implemented the first aspect of structure morphism: RELATION::Grow, and TYPE::Grow. This is crucial milestone of the ANNA K5 core development. Obsoleted TBaseHierarchy::Size as a consequence. Defined more clearly the notion of Background relationships. These describe the direct inheritance chain that’s necessary in IsType and DescendsFrom calls throughout the core. The background relationship needs to be first and not moveable. This relationship holds the default (design time) size of the type. GetSize calculates the type size based on the sum of all relations and maximum against the Background. TYPE::SetAncestor(ancestor,size) now builds a Background relation configured with size as Cardinality.

Enhanced the TREE::Insert method with a flags parameter that may be set to REPLACE occurences of the same name along the tree. Added the Replace method along the TREE hierarchy to handle replacement. This is a subtle algorithm. Now we can load the design from a MDL or BIN file and drag-drop them using Insert(REPLACE) over the default ABSTRACT category to add the symbols to the tree. This is an important result that makes the automated log and stack viewer possible.

Corrected the TREE::FindCreate method. The Rose model loader is now using it to create forward declarations. The mechanism is critical in finding or creating tree items along a path. This method has resolved the problem of creating nested scopes. Re-worked the CreateInstance and SetTypeMaster mechanisms. More effort needs to go into default constructors for all internal types to replace the SetTypeMaster functions.

Enhanced the Rose loader to understand enums and create constant clusters for them.

Changes to NODE/LINE drawing and correlating. Draw and correlate are aspects of the same thing: serializing over bi-dimensional streams. The abstract CANVAS stream was introduced along with a new generalized Serialize method. Drawing and correlating are now done in the order of refference. Aided by the SerializerCache field, the method only outputs an item once. Consequently drawing is much faster.

Correlate is obsolete. The new Serialize method should soon absorb the old one as well as the Load and Store methods. Drawing is now driven from the core and simplifies the GUI. The ability to highlight selected items was lost and needs a thorough re-think with the use of another tree aspect for nodes.

A new function was added to TREE to speed-up the frequent tests for a line ending. TREE::IsLineEndpoint uses forward knowledge of a Line’s adiacent pair of tree nodes that make-up the endpoints and looks for them without calling IsType. That was a recursion difficult to close. Navigation has become significantly faster as a result of avoiding IsType calls.

Added intelligence to moving nodes to “absorb” degenerated line angles.

Nicer bitmaps for TypeMasters. A vector of type masters maintained by TBaseHierarchy to quickly look-up class IDs in FindByClassID calls made by the binary serializer.

Added GUI logic to list the class properties in the parameter sheet along with the actual values.

Nicer bitmaps for TypeMasters.


48 Sunday, June 10, 2001

Drastic cleanup of the free-floating variables and functions. Merged ABSTHELP.CPP back into ABSTRACT.CPP.

Modelled the bit memory shifter functions from Rose. Included the code sections of several functions into the Entry/ExitCode sections of the Rose Model.

Enhanced the Rose model loader to create a node with Information set to EntryCode and another set to the ExitCode strings.

Corrected a problem in the model loader that affected the loading of class attributes and consequently was not creating constant nodes.


49 Thursday, June 14, 2001

Changed NODE::Expand to create NODE_CLIENT nodes for lines rather than for the To nodes of a LINE.

Sped-up the GUI initialization by using the new TBaseHierarchy::TypeMastersVector to collect IDs and bitmaps.

Several fixes to the mechanism of destroying NODE_CLIENT when NODEs are deleted.

Fixes to the Insert(t,REPLACE) algorithm.


50 Wednesday, July 04, 2001

Using the serializer to find items. Enhanced the Serialize methods to return an OBJECT which is the result of the find.

Fundamentally changed the Serializer algorithm: pointer the relocated pointers to serialized instances are now relative to the independent chain and not to the type master instance lists. The mechanics of SerializePObj and Mark are quite subtle. STREAM::Markers and STREAM::MarkersCount are two new fields that achieve concurrent, independent, and complete chains in a pointer map. The stream destructor is now in charge of clearing the OBJECT::SerializerCache chain.

Objects are serialized without ever seeking over a stream. This mechanism makes it possible to serialize a map onto a queue. The new scheme has it so that there needn’t be a relationship between the stream and the serializing object such as using GetPos as a marker. To successfully serialize all we need is to mark the fact that a particular instance has already been serialized. Additionally we need to chain the serialized instances only so that we can find and clear the markers once serialization completes.

STREAM::SerializePObj returns a generic object pointer that the Serialize methods collect. This mechanism is now used to correlate inside pointer maps. SerializePObj is now independent of the operation performed, acting much like an iterator over the sequenced pointer map.

Several corrections were needed across the system to support the serialization of compound objects. The default constructors were re-worked.

Serialization of LINEs has been split-up across LINE::Serialize, ::SerializeLabel, SerializeFrom and SerializeTo, so that RELATIONs may override them. Better drawing of lines and nodes with special circle symbols at the points of intersecting with the endpoint nodes. The new drawing routines work for any polyline shaped nodes.

There is no difference between serializing to a file or to a bitmap and all STREAM helper methods were introduced to serve both types of specialized streams. An example is STREAM::Ellipse that defines the abstraction but does not resolve the actual drawing for a non-bitmap stream. The code generator now produces a circular script, for whomever can read and write like that. This is the effect of generating both script and drawing with the same code path. One specialization had to be made between serializing full pointer closures to binary files as opposed to serialized them to files or display streams: serializing to a file or bitmap must not flow out of the selected scope. This means that we only want to render a branch of a tree, and not its whole ancestry and siblings. This principle needs to be re-visited so that the specialization of EFilter::NODES disappears and is replaced by a system of selecting a closure of import nodes.

All in one this version is an important step forward.


51 Friday, July 06, 2001

Corrected the Type Master reporter to show the last element.

Creating an Enterprise test structure in the initialization of the form to have a node other than MORPH to drop into.

Developed new logic to drag from the ScopeView and drop nodes into the Schematic. This mechanism creates new instances of the dragged type.

Re-instated the relationship wiring tool between nodes.

Corrected the ability to change a node’s name from the tree viewers.

More careful serialization of the secondary pointers of an Enterprise.


52 Sunday, July 22, 2001

Preparation work for auto binding lines.

Added serializer support for external nodes.

An enhancement to FindCreate to help serialize external nodes in.

Changed the order of construction inside the category constructors in expectation of the new line auto-bind mechanism.

Added the cable argument to the Serializer functions.

Removed the VAttribute structure and merged it into the NODE class.

Corrections to inversion of matrixes.


53 Thursday, August 02, 2001

Introduced the cable principle to keep lines together.

This version draws the diagrams from the perspective of the null line of the current node.

The detail level is tuned on the selected node rather than the root.

Serializing the TREE parent first to affect the draw order and have less children drawn behind their parent.

GetLineEndpoint is a new method to return the line endpoint position in the coordinates of the caller.

Introduced the cable principle to keep lines together.


54 Tuesday, August 28, 2001

Inserting the generated category into the ROSE category was establishing a parasite line between the two categories. The Gen category was also saving the entire MORPH.

Changed TREE::ParentIs to return true if this==that. In other words, a component is at least its own parent (for now). This is to auto bind the lines of the Enterprise.

Changes to the Serializer to rely less on the NODES flag.

Corrected a destruction problem with grammars pointed to by Nodes.

Corrections to FindCreate to resolve paths and create new instances of the type passed as argument.

TREE::Serialize is now responsible for umbrella clipping and not recursing above the umbrella.

Hierarchies have a Square shape by default.

Line autobinding have complicated the constructor of the core and the order of binding. Needed a shortcut test in GetFirsRELATION.

RELATION::Remove shouldn’t have reset the origin. The new line autobind mechanism may remove and re-insert lines and relations, and doesn’t want to loose the Origin field.

Vacillating between serializing the contained bits of a category or not.

Cleaned-up old Enterprise traversal code that has falen behind and is now superseded.

An important fix to the recursion of GetCommonParent: the former version returned the oldest grandparent, the new one returns the closest parent. This change was necessary to correct the line auto-bind mechanism.

Designed the SpiralTile function that keeps children together but spaced out using edge adherence logic.

Renounced the Import and Internal NODE data members. They are now emulated with a better line auto-binding algorithm.

Severe changes to the LINE and NODE serializers.

Added another level of complexity to GetLineEndpoint to go up the parent cables and involve the cable endpoints into the calculations.

Lines can auto-bind and auto-remove.

Corrections to the serialization of exports. Removed the detail clipping from the responsibilities of STREAM.


55 Monday, September 03, 2001

Changed TREE::ParentIs to return the depth. If this==that the function returns 1. If that isn’t a parent of this the function returns 0.

TREE::Serialize is flat again and it is not responsible for umbrella clipping any more.

GetLineEndpoint calculates the intersections in the coordinates of the common parent of from and to nodes. Simplifications to NODE and LINE serializers.

Achieved correctness throughout all world transformations, detail clipping and cabling. :Lines are back to being responsible for drawing their two arms.


56 Tuesday, September 11, 2001

Changed the serializer yet again: nodes serialize flatly, lines are in charge of serializing their endpoints, they both override the TREE serializer and never call it. Calling the inherited TREE serializer was holding me back. Introduced icons for relation endpoints: diamonds, circles, arrows. Lines are serialized in their normalized coordinates.


57 What happened here???


58 Monday, February 25, 2002

Changes to MORPH destructor. Created the"workspace" node inside MORPH for users to store their stuff. Changes to the Serializer: the direction flags include the notion of Canvas. Added another flag for relations to denote an INSTANCE relation. An instance relation is drawin by cyrcling the immediate public children of the node. Added the MakeCache function to the NODE class to recurese the drawing method and paint in a local canvas. Enhanced clipping of lines and Ellipses viewed as rectangles.

Added the grammar based serializers needed to load the morphing assembler scripts. Added a few more methods to TDatabase to create a DB, create and remove a table. Enhanced SQL error reporting. Added more to the database self test. Creating a database isn't working yet. Possible cause is the CREATE DATABASE statement needs SQL server to work. Added the ability to FindCreate RELATIONs. Added the Step and Expand functions to NODE to create the intelligent logging system with Stack, Coverage and Models sections. Step is used to increment the counters of any NODE_CLIENT attached to the node. Step works recursivelly up the tree. Expand is used to either expand or collapse any NODE_CLIENT of the node. An endless recursion was entered when adding a field to a structure that had a static instance of itself. RELATION::Grow was not taking into account the static attribute. Collected all the core tests into k5core. Corrections so that the Help file gets generated with no errors. Turned Saturn TSuperKernel into an RPC server.


59 Wednesday, March 06, 2002

Added a GUI option on the node editor’s local menu to send the selected node to the back of the deck. The same could be achieved by dragging and dropping the node into its parent, but then the auto-replace mechanism would be invoked and would reduce a node with the same name. This is the case for most lines and cables that are unnamed.

Promoted the REI driver and the Debug Server to Kernel. ANNA K5 is now the debug server. Relinquished the model. Regenerated the sources from the model.

Caught a fault in the Remove logic that was not updating the NODE_CLIENT chains, thus affecting the ScopeView.

Re-worked the Schematic Editor to have two Rebar controls for nodes and lines. Changed the mechanism to load and edit bit fields to be more generic. The line editor includes a field for editing the cardinal of the relation. The concept of cardinality needs to be promoted to NODE to help with matters such as licensing and anything else that says “there can be only X number of connections to this node. Enhanced the GUI to draw the CurrentNode to a bitmap that goes into the clipboard. This step greatly helps with my documenting effort.

Corrections to the Rose Model loader. Once the type masters are recognized during the first load of a model, they will be re-used in subsequent loads.

Enhanced the model compiler to accept duplicate field names that arise with interaction diagrams. The relation driven Read/WriteBuf returns an error if no relation with the specified name could be found, and does not alter the value that is passed through. Must initialize the values passed through ReadBuf.

Node auto-centering on a tile operation. Corrections to the GetCommonParent function. Changed the way in which the line AUTOBIND mechanism ascends towards the common parent of a line’s endpoints. This has improved the coherence of complex, multilevel diagrams. It seems that the new style of cables are organizing themselves in a top-down fashion, more suitable for a future top-down serializer. The new auto-bind has also corrected the problem of intersecting lines and nodes that were transformed by rotation and zoom. Promoting the parent pointers through constructors to do line autobinding for all core components.

Corrected the line pick correlation on the “to” side. Added the concept of weight for cables. This is a recursive increment on the fact that the line has children.

This version allows the dropping of instances of types from the dictionary onto the diagram. The relation is has-by-reference at first with a visual flag of HideLines and one of Instance. A relation viewed as Instance would recurse the Serializer to draw or pick correlate the node at the “to” endpoint of the relation. This version of the Enterprise Kernel of ANNA K5 allows recursion in diagrams, or the Fractal Containment declared in the Manifest. The pick-correlation of instances is now automatically creating Selectors. Selectors are hidden constructs that alias a node or a path of nodes and bind to the containment relation. This version of the kernel uses the generic expression to link two selectors together and acheive wiring on the diagram. This is the beginning of semantic evaluation at the core of ANNA. Moving a node was changed so that line auto-straighten is not attempted for Instance relations which are generally carefully positions by the user. Changed the serializer to accept the NOLINES flag and use it to recursively draw a reference to a node with everything the node has except drawing the lines.

Corrected a problem destroying grammar nodes.

Corrected a few problems to do with direction in a few methods of stream.


60 Wednesday, March 13, 2002

A change to the stream serializer to load and store export nodes.

Added the SerializerInsulation and the Exclude functions that mark the frontier of a tree as being external.

Added the AutoLines method to destroy and re-create cables whenever a node is promoted or demoted in the scope hierarchy.


61 Thursday, March 14, 2002

Version IDs embedded into the Enterprise binary files. These are the VersionIDs of the type masters that created the instances.

Simplified the Serializer parameter list. Corrected a destruction problem of Morph classes.


62 Tuesday, July 23, 2002

GUI changes: Threaded loader configurable from the Options tab. Reinstated the Runtime Bridge. Charts and CheckBox viewers. Added logic to protect the TypeMaster viewer when new types have no associated bitmaps. Enhanced the drag-drop logic that creates instances to also position the new instance at the mouse position. The new has relation is a child of the current node. Corrected the Information editor. Refreshing the Type Masters list whenever the tab is shown. Changes to the rotation logic.

Correction to the binary serializer to avoid recursing over the Next and Prev pointers of the current node.

The current node is stored with its Next and Prev pointing to itself, as if it had no siblings.

NODE.Colours is a point type.

Enhanced the point class with the length method.

Brought the TRuntime class into the ABSTRACT category. Added the RuntimeAttach capabilities to the NODE class. Added a node method to clear unused selectors. Colours, V_Size and V_Origin are now points. A few dummy fields were needed to pad the 2D points from the former 3D. Introduced the Angle field as int. Enhanced LINE::FindCreate to take the name string as well.

Sophisticated logic to create and then fix selectors to fields when the design of the referenced components change. Selectors are fixed but not drawn, they are invisible in this version.

I don’t know why we now store to binary the (content of) instances of a type. Need rethink.

Better logic to calculate line endpoints with recursive cables that works top-down.

GetLineEndpoint is obsolete in the top-down drawing algorithm.

Caught an error to do with the FIND_CREATE flag.

Changed the Serializer to draw top-down on a CANVAS. This refinement has sped-up the drawing process by 100 fold. Its importance is in that a clipping engine is now possible. I’m working towards merging in ANNA K4’s clipping engine.

Need to protect the container pointers from serializing a second time.

Caught a problem in the Core constructor that was creating a new instance of the REI Agent instead of connecting to the existing one. Morphism core could not be shared. ConstructTInterface does not need to return a pointer.

Added the Bool type master. Binding “void”, “char”, “int” and “bool” to the ABSTRACT category.

Added the Instrument Server category. ICam types are now TYPE descendents. The unique IDs of the Instrument server are given by the abstract core.

Tunned a few exports.


63 Thursday, August 01, 2002

Added a 1s timer that looks for MORPH::Changed to invalidate the Schematic Editor, if the backbone was changed via the RemoteExecInterface.

Added a static Changed member to the MORPH category as a crude way of anouncing that the back-bone has changed and Navigators should refresh.

Added code to the interface driver to get the application path and build the INI file from there. The path isn't yet used.

Included the marshaling macros.

Added functions to the TRemoteExecInterface class: GetNodeParent, GetNodeNext, SwapColours, besides the GetRootNode, GetSetNodeName, and GetNodeChildren.

Implemented most of the interface functions enough to get remote browsing capabilities.

Changed GUI to use scrollbars to edit the node Angle. Added a system colour picker.

Allowing the draw of lines in referenced nodes.

Implemented the isoscellous trapeze node face.

Added NODE logic to render bitmaps, JPEGs and metafiles.

Added a WriteBitmap method to STREAM. A Canvas descendent overrides this to implement.

TRemoteExecInterface is a new class in the morphism core that exports via REI a few functions that another application may use to browse an Enterprise structure.

TSuperKernel class was promoted from SATURN to MORPH.

TSuperKernel ofverrides these abstract REI methods to implement the calls.


64 Wednesday, August 07, 2002

Registered the STREAMS category with the MORPH category.

TCanvasStream was renamed from TAnnaCanvas and was promoted from GUI to the Morphism Core. Theoretically a MSVC build is possible with schematic capabilities.

Reworked the TAnnaCanvas class to achieve CBuilder independence. Lost JPEG, WMF and transparent bitmap support as a consequence.

Caught a problem with text nodes that returned a 0 points shape to the line intersection routines.

Removed dependencies on the node matrix which is only kept for binary file compatibility.


65 Tuesday, August 13, 2002

The Core can draw mirrored pictures, given the new pick correlation logic.

Promoted the CreateLine logic from GUI to Core.

Added size information to the GUI status-bar.

Fixed the problem of diminishing the detail level of nodes that was leaving wires of greater detail visible.

Resolved the visuals of lines that bind to round pins.

Changed the LINE::PointOnLine correlator to use a circle as the selection point.

Fixed the problem of drawing lines over the sub-nodes.

Changed the Node serializer correlation routine to be exact with respect to the node shape.

Enhanced the LineAndSegment intersection routine to return the following codes: 1 if [AB]^[MP] falls within both segments, 2 if I if on [AB

3 if I is on AB, and 4 if the line doesn't intersect [MP] between M and P. These new codes are used now by the new HitTestPoly function which is an exact correlator of polygonals.

Added two new functions to calculate a normal Ellipse intersecting a line, and a general circle intersecting a line.

Added the STREAM::GetSelectionRect as an abstract channel through which the Core may correlate. Added the CurrentMatrix to the CANVAS stream to ease mapping operations without going through DPtoLP, LPtoDP etc.


66 Thursday, August 15, 2002

The visual flags have changed, so binary file format has changed.

Enabling Shape for relations. Lines thicker than1 cannot be dotted without a geometric pen.

Lost the TREE::FIND_EXACT=0x10 flag.

Removing the matrix from NODE.

Resizing the node through a GrowBy method.

Added the Tileable flag for node visuals. Added the Visibility enumerator for keeping nodes private to the design.

Changed some flag IDs. Removed the vaUnderline and replaced it by vaShowName.

Corrected the matter of setting the foreground colour for text.

Added UserData to NODE. It's used by bitmap nodes to hold the bitmap handle.

Loading bitmaps when serializing the enterprise in from a binary file.

GUI changes:

Better Schematic Editor toolbar with editor for node visibility, for line recursion, and node visual flags.

Changed the ColorGrid to export its palette table. Using that instead of our own. Resolved the editing of this palette through the system colour picker.

Changed the statusbar to show the selected node's X,Y,SIZE, the current mouse coordinates in the context of the canvas stream.


67 Friday, August 23, 2002

Implemented the semantic validation on wiring in MORPH::CreateLine. A new method TYPE::IsAssignmentCompatible that compares two types and looks for assignment relations between them or between their ancestors.

Changed TYPE::DescendsFrom function to look for indirect inheritance as well.

Changed the destructor of a relation to first shrink the relation to 0 size before actually removing it. This is for morphing the From endpoint of a deleting relation. Mechanissm is used by the schematic editor to allow deletion of assignment operators in fundamental pin types.

Changed the semantics of a Selector line to lay above the field. this lead to the removal of all cables in a wiring diagram, which is a significant improvement. In other words, the code of a module is flat. The algorithm that resizes/repositions the selectors whenever the underlying instance is moved/resized had to change.

SerializeTo and From were replaced by SerializeEndpoint that only deals with arrows. The line itself is now drawn by a recursive SerializeSegment function that takes into account the bend points.

Heavy changes to all Serialize functions. Return from the function needed to be enhanced to not only return an object pointer but also two more ints about the object. The new scheme is used by the Line multi-bend algorithm. Added NODE::ESelectionCodes to extend the serializer with more info about the selected node.

Enhanced the NODE Flags with vaClip, vaUngroup and vaSelector flags. vaClip is used to limit the mouse correlation to within a node's bounds before recursing the children. vaUngroup is used by both nodes and cables to free their children into their parent. vaSelector isn't used yet.

Class point is now controlled. All correlation logic is about to move to its methods.

Enhanced the LineAndCircle function to return all possible intersection cases.

Enhanced the LineAndEllipse to take the origin as a parameter.

Loading a bitmap node is now done upon the first time is drawn, to allow us to change the name and the bitmap. A bitmap node needed to return a rectangle shape.

Added all necessary logic to highlight the current object. This is the inverse of the correlation process.

Needed to change the MoveBy method for both NODE and LINE and have it return a code to inform a GUI that re-correlation is necessary. This is due to the artificial intelligence of bending lines that may create or remove breaking points as a result of this call. A GUI would then need to correlate the newly created item.

Improved the midpoint absorbtion algorithm.

Corrections to coordinate transformations. Get and SetPosition now take coordinates into account. ConvertCoordinates now takes two flags: from/to interior to further qualify the translation.

Removed all dummy fields of NODE.

Pointer arithmetic spawned from TBaseHierarchy to deal with object layout shifts from one version to another of the Enterprise binary files. Current ANNA_VERSION is 4 as published in UIDS.H. These functions are virtual static.

All classes should implement a virtual Test method their constructor would call.

Caught a bug to do with the size of CanvasStream typemaster.

Enhanced the Canvas stream to sense a pre-2000 host OS and emulate the world transformations.

GUI changes to the Navigator: adapting to the new serialization logic of the core. Keeping track of the latest two user selections that may get wired together. Added the Clip and Ungroup flags. Added the relative XY position to the status bar.

Added the file system version to the About tab. Deleting a node doesn't need to reload the TypeMaster tab. Logic to preserve the correlation information and pass it from the paint method into the drawing serialization of the Core. Re-correlation after moving and possibly breaking, a line. Dropping nodes under the mouse pointer onto the diagram. Allowing an instance to be dropped onto the diagram should only work for diagrams of a hierarchy. GUI doesn't want to know about Selector lines. Changed the actions of the application. Changed DEL for CTRL+DEL and INS for CTRL+INS.


68 Thursday, August 29, 2002

Changed the K5Test executable to use hardwired paths to the rose model files. This is due to the nesting function of STREAM that cannot match relative with absolute paths to the same file. Enhanced the console with a command to turn STREAM debugging on and off.

A better MORPH::CreateLine function that looks at all cases of from versus to nodes.

Cleaned-up the initialization of hierarchies that are their own type.

Consolidated the GetVMT declaration and definition using DECLGETVMT and MAKEGETVMT macros.

Fix to HitTestPoly to use the center of the polygon as the semi-line start.

An attempt to follow the class layout of RELATION descending from LINE and TYPE, both descending from NODE as a virtual base. Needed to enhance TYPE to hold the BCT_NORMAL as the delta from class to virtual base. Two new NODE functions LineUpcast and HierarchyUpCast convert a NODE base pointer to the derived class pointer using the TYPE BCT_NORMAL offset. An attempt was made to make RELATION descend from TYPE and TYPE descend from LINE. A new NODE::IsLine method was required along the hierarchy to tell the difference between types that descend from LINE for implementation rather than conceptual reasons.

Using reinterpret_cast to force casts along virtual inheritance.

Drastically reduced the INSTSERV category to only the display components.

Removed any unnecessary casts to TYPE* left over from TBaseHierarchy days.

PointerArithmetic dispatcher need to move from TYPE to somewhere below the NODE virtual base to eliminate ambiguity. It moved to OBJECT.

Caught a bug in the Rose model loader that crashed with supplier_cardinality undefined.

Removed the Comment field from RELATION and TYPE. Moved all the intersection routines from PROCESS to MATRIX\point. Create a user type and a morphing instance of that type. ANNA K5 can now instantiate the design by allowing the user to create has relationships in this user type. Merged TBaseHierarchy and TYPE. Removed the ENTERPRISE class. Its role has long been replaced by MORPH.


69 Sunday, September 15, 2002

The heaviest changes since Rev 1.20 of the ABSTRACT in 21 Feb 1999 17:34:08 Alexander_Mihail_ALFA ( The semantics of the language have changed. Functions and relations are now classes):

Rationale: Lines are instances of Relations. User relations are also relations. Deleting a user relation deletes all wires of that relation. Changing the attributes of a relation affects all wires/lines of that relation.

1. Relations are now both Lines and Hierarchies by double descent. The major problem of unique instance identity for lines needed to be solved. A relation can only belong to one instance chain of a Hierarchy. It follows that a relation cannot have two NODE ancestors as it would directly inherit from LINE and from HIERARCHY. Therefore LINE and Hierarchy had to descend from NODE virtually so that Relation only ever gets one NODE aspect for visuals and for chaining instances.

2. The Autobind mechanism has changed to create structured assignment operators whenever any containment relations are wired between two types. This greatly helps the understanding effort a user needs to put into the design of components.

3. Changed the construction scheme altogether to cater for multiple inheritance with virtual base classes. Changes were done in two passes: first rebuild the whole lot with the new style of inheritance and with forced casts through void* to get it done. Secondly, I had to rework the entire construction scheme to cater for 1. double inheritance, and 2. for virtual bases. The old GetVMT strategy was insufficient to gather all the aspects of a BC5 class layout with virtual bases. new TYPE fields are VMT_STUB to hold the stubs table of the virtual base, BCT_NORMAL to hold the offset from the instance top to the virtual base aspect, and VMT_STUBR that records the second stub table for the second inherited aspect of a double inheritance arrangement such as RELATION. The following logic needed to change: OBJECT constructor/destructor and SetTypeMaster/UnsetTypeMaster to setup those fields in the instance data. CONSTRUCT/DESTRUCT macros have changed. All ::Type pivots are now captured upon the default construction of the components of the SuperKernel. Once gathered, all these type masters are partly initialized in terms of their VMT and BCT fields in the SuperKernel constructor before MORPH::INIT is called.

4. The matter of casting pointers across a multiple virtual inheritance required special cast functions. We may have a pointer to the instance of a specialized type and want a pointer to the virtual base aspect. OBJECT::Cast does that. Or, we may have a pointer to a certain aspect of an instance say to the LINE aspect of a RELATION, and want a pointer to the Hierarchy aspect of RELATION. OBJECT::Cast2H does these necessary offset adjustments. These casts are wrapped into the morphism_cast(type,data) macro.

5. RELATION is a Type pool so that user relations are RELATION instances and are also relations. Serializing a RELATION posed the problem of double serialization of the NODE aspect. IThis was resolved with the VirtualBaseSerialized field that gets set/queried by LINE::Serialize and by NODE::Serialize.

6. Added the SelfTest methods to all classes to check for integrity. This new method looks for binding problems and was an important debugging tool in this development process.

7. Extended the binary Serializor with relative paths of exports/imports recorded and decoded once only. Also, the FindCreate and AggregateRelativePath functions that manage these relative paths to and from a node and its environment were extended to account for the (frequent) duplication and noname situations. by coding/decoding the duplicate occurrence ID along the path. These functions use '.' as filed selector and conflicted with the "." name for selector lines. Selector lines are now labeled with "$". The Serialize methods needed to return a boolean to tell the caller that an export/import was found and special action may bee needed such as inserting into an import parent. These are hugely complex matters regarding the subdividing of a referential closure. Added the"down" argument to FindCreate to allow the algorithm to recurse upwards into the scope looking to match the given path pattern. This is the old forward declaration resolution mechanism. One more extension is that FindCreate now creates lines that are end-pointed upon their scope, since semantic correctness pleads that lines must never be unbound in the healthy system.

8. Serializing a tree with exports has gotten complex since trees are abstract and they will always be contained in other type instances. We can expect any combination of exports on the Parent, Next, Prev and Children pointers, resulting upon loading on several requirements of re-insertions.

9. STREAM::SerializeExport is new and delicate. It gets called through the SERIALIZE_EPOBJ macro on any pointer that the Morphing OS suspects to be lying outside or above the umbrella pointer, or is beyond the nominated depth. This function serializes a marker to say that whether the node is an export, calculates the relative path from pointer source to destination, and reads it back on load, to call FindCreate. It thus matches exports/imports upon load or it re-creates the missing branches of the path with defaults. SerializePObj returns boolean to express the export/import state of the operation.

10. NODE::Serialize is much cleaner now as it doesn't deal with umbrella clipping anymore. It calls TREE::Serialize.

11. Added a Hexagon shape to NODE.

12. PointerArithmetic has changed to take the phase argument, instead of relying on a null this pointer to dispatch. Calculating pointers from previous version layouts has now become difficult, if not inoperable, with multiple and virtual inheritance. I am hoping to obsolete this mechanism in favor of morphing storage.

13. I never accepted the C++ constructors/destructors that gradually promote an instance from base type to descendent type during construction and then demote the instance to base type during destruction. The morphing core needs the ability to call virtual methods accurately any time during construction or destruction. I'm therefore in conflict with the delete mechanism that wrongly deletes our instances of multiply virtually inheriting types. To alleviate, OBJECT destructor does an explicit free of the instance by testing and setting the hidden argument.

14. Attaching a GUI to a Morphing XRef has changed. TRuntime is obsolete. STREAM::GuiAttach is new and used by the serializer of RELATION before recursing the "to" endpoint. Enhanced the system to render a morphing cross reference to a runtime window. Treating the runtime window as a stream has allowed the reuse of the Serialize function to create the nested trace windows of a type. Obsoleted the INSTSRV category. Any type may have one or more dedicated event viewers and event generators. The role of INSTSERV was to interpose instances of special types along the wiring scheme to achieve data tracing. This mechanism of tracing/pumping is now generalized for all relations through the new Relation fields ViewerType and ViewerBounds.

15. GUI changes in the Navigator: Reworked Trace, CheckBox and introduced Generator forms. Made these forms to descend from TForm_RuntimeWindow that has all the logic of constructing/destructing attaching and detaching in response to TRuntimeStream::GuiAttach calls while tacking care of the delicate matter of relation offsets into the data instance. No need to isolate a tree before serializing it because of the new export/import scheme. A new combo box was needed to select an event viewer or a generator type for a relation. Simplified the configuration of flags for Relations. disabled all unused flags. Changed/simplified/corrected the loading/storing/canceling of the toolbar content. Removed any unused event handlers. Using a much more interesting method of loading a binary XRef, one that has deeper implications into the future of the system, its autoload function, and its file-system with parts loaded, parts not: to load an XRef we select a node and a filename, then the STREAM gets created as a child node of our selection, it gets loaded in a way that satisfies the export/import relative paths and sibling to the STREAM node. then the stream gets destroyed leaving only the XRef. Added the beginning of a keyboard handler for the schematic editor.

16. The Enterprise Evaluator was implemented and has been seen working. New parameters are: the data pointer that permeates through, the offset into the data buffer that keeps changing at every hop, and the length of data requested for update/evaluate. Evaluating a node means to shift data from all input lines of that node into the node. A new LINE::CopyData function was devised for this data shifting. Granularity is bit. LINE::GetSelector endpoint was extended with an argument that calculates the offset from the container type to the endpoint relation to help the evaluator to find the relative offsets before data shift. The Evaluator starts from a THREAD structure by finding the thread's inputs and evaluating them. A thread establishes the fundamental evaluation order in a morphing xref.

17. All runtime windows are animated by the runtime thread or timer that looks them up following their pivot chain.

18. A test of a LINE for actually being a line is now done by comparing the VMT.

19. NODE::SerializeFace is a new way of allowing an overriding Relation to recursively draw the supplier's face.

20. Line segments now inherit their colours from their relation type master. This allows a user to change the attributes of all lines of a given relation centrally, at the relation definition point.

21. RELATION::set_Flags and set_Cardinality were needed to trigger structural morphism.

22. CreateNewInstance over a char buffer is a new way of constructing a (default) class instance over a controlled buffer, so inside a layout, by cooperating with the override new function. This mechanism was quickly abandoned in favor of a more direct way of gathering VMT and displacement information at core startup.

23. TExpression is also a type pool, like RELATION. Expressions generate other expressions.

24. TYPE now serializes its instances as well. It was needed to properly bind the wires generated by relations.

25. USER category is a placeholder where the user may safely trigger morphisms without getting the operating system to shift. MORPH::FileIo is new and encapsulates logic to stream in and out a morphing xref. Enhanced the MORPH::CreateLine case grid with THREADs and with constructing wires as instances of the relations uniting the type endpoints of the selectors involved. MORPH::CreateNode is the new response of the Core to GUI drag and drop. A new special case of drop here is creating instances of threads.

26. TSuperKernel doen't descend from MORPH, it rather contains it.

27. K5CORE has a new test for fileio to serialize in and out xrefs that may be returned by main_parser.

Strange non-completion return codes from ::read, probably due to threaded shared files. The API error handler didn't return anything. Caught an uninitialized Database type that caused a corruption in Pivots. Canvas stream didn't have its own Type pivot, leading to a corruption. Prettier bitmaps. The rect class is now controlled. Some classes "have" it by value.

Added two more cases of wiring involving user types.

Changed the AutoBind mechanism to auto create assignment relations whenever lines are bound to types.

Added the OBJECT::TestType hierarchy discriminator function.


70 Tuesday, October 15, 2002

GUI changes: StatusBar replaced by a panel with buttons. Added a Log tab. Added the XRef status display. Threaded evaluator and GUI refresh.

The battle for exports continues: Added the STREAM::IsExport function that people may call to take their special serialization action. STREAM::SerializeExport now uses the IsExport function to test the umbrella.

Corrections to RELATION::SerializeFace. Corrections to RELATION::GetSize and TYPE::GetSize to make it past the operator relations.

The Evaluator now works. Added the Evaluate and CopyData methods to RELATION.

TExpression moved from ABSTRACT to COMPILE. The drawback of the move is that Cast2H cannot work on expressions and their descendents now.

Int, Char, Void Bool moved from ROSE to ABSTRACT. Inter operators for Int and Bool.

TYPE::DELTA_SECOND_ANCESTOR was left un-initialized.

GetXRefDegree is new and used to show how many elements depend on this one.

Reintroduced the InstServ category. More complex logic for Displays and Generators.

Added logic to trap the core's test messages to a GUI text panel.

The USER category needed to be a hierarchy descendent.

Better FileIO logic that takes the share mode into account.


71 Wednesday, January 29, 2003

Re-archived on Independence\core after having lost 2 revisions on Carmen\Core.


72 A pause of a few months for some severe thinking.

LINE::PointOnLine was missing one important intersection case: the segment (5), making correlation difficult at best.


73 Wednesday, March 19, 2003

Increased the stack size. Enhanced GUI with Step daemon.

Changes to the model files to reduce the forwarded symbols (Ex: char**).

Implemented the MEMORY_STREAM class.

Found a way to reduce compiler backtracking by rotating the loaders in the list based on frequency of use.

Calculating label bounds for relations.

Corrected a pick correlation problem due to the wrong clearing of the selection in LABEL::Serialize and TConstant::Serialize

Reorganized the command line k5core utility.

Corrections to the autobinding mechanism to create assignment operators strictly for wires. Allowing now the formation of super-cables between nodes that nest types.

Resolved a crash condition due to violating morphism casts inside LINE::GetWidth

point::SegmentAndSegment returns the t parameter between the lines that determine the line. Some callers such as LineAndPoly were interested in choosing the smallest t.

point::distance is new. LineAndPoly2 is new and returns the intersections in the same point buffer where the poly is described. MinSegment calculates the minimum segment between a pair of sets of points. It is used for the new line midpoint calculations in NODE::Tile.

Corrected a rejection condition in SegmentAndSegment that was not clipping vertical lines due to a zero determinant.

Enhanced the TREE::Find algorithm with another flag to do exact matches on types. Otherwise the function is now looking for any compatible type.

LINE::Exclude is now obsolete.

Corrections to LINE::SerializeSegment to do with lines between a scope and its (grand)parent. Corrected a missing matrix restauration.

A fix to LINE::ReduceBends to do with lines between a scope and its (grand)parent.

The Rose model loader was creating errant relations to other relations when the Find (inexact) returned a class constructor relation rather than the class type.

Renamed constructors and their prototypes to _Name.

Inserting and removing nodes will step the target.

Changed the tiling algorithm: first we tile the nodes until their new positions settles, then we tile the lines. We work with children and their lines in the context of the current node, without changing coordinates. In the end we use the new point::MinSegment logic to get the midpoint for lines.

We now show the labels of sub-cores when the cable ungroups its children.

FIXME!: Don't know why is EMoveBy being promoted to the top category.

Caching file streams with memory streams to speed-up the rose compiler.

Introduced another aggregation scenario through USES relations. Uses relations are thought of as proxies to their supplier or client type. Any relations between them actually apply to their endpoint types. They represent "funnels" of XRef. Uses relations have no size. get_Supplier is now more complex through uses relations. Changed the calculation of MaxConstant.

Passing the features flag to a TCanvasStream constructor to give the caller a chance to disable scaling, which is not always supported.

Enhanced the GUI with an option flag to disable WorldTransform and produce EMF files compatible with Corel.

Added GUI logic to clear sizes and positions of nodes. Added GUI logic to show "daemon" activity indicators for nodes in the browser, as scopes step onward.

Corrections to the GetName and Information routines to take the buffer length into account and stop crashing on long/deep names.

Detail is configured for the Current node not for the selected one.

Fixed an interesting crash condition when loading model files. The stream had to be destroyed before inserting the branch into the global enterprise.


74 Tuesday, March 25, 2003

The file format has changed to only store the bend buffers for lines that do have bends.

Pick correlation was wrong because CANVAS::MoveTo and GetPos were going through the device context with world transforms. I believe (without being too sure) that repeated LP2DP2LP2DP conversions were lossy. The canvas now keeps the current position in a point field, and returns it translated once through its managed transformation matrix.

PointOnLine now returns a hit if all three points are the same.

More fixes to the Midpoint Absorbtion logic. Relations from a node to its (grand) parent are now absorbing well. Resolved the errors of choosing far away intersection points by using the new MidSegment routine. A difficult fix to drawing of cable sub-cores at depth level greater than 2. Corrections to logic that draws sub-lines with bend points, when the cable un-groups them.

Fix to the matter of bending lines by moving a segment with the mouse, which used to create heretic points.

Increased the size of the array of points passed to GetShapePoints. It was set to 5 although the array had 10.


75 Thursday, April 03, 2003

Simplified code with a NODE::IsSelector function. Added a NODE::AutoBindEnabled static flag that GUI may turn off, to study the effects on diagrams.

A change in the NODE serializor to allow a line to show its internals when selected as the current node in the GUI.

Improved GUI logic to filter information about nodes being deleted and deselect the current node if necessary. Corrected the multi-form logic by establishing a chain of forms. Corrected another bug/crash to do with long tree text. Fixed the Navigator's title and icon.


76 Friday, April 04, 2003

Enhanced the Canvas initialize with a delta view-port factor. This was needed for serializing into a smooth scrolling bitmap.

Enhanced the GUI Navigator to render the schematic onto a (potentially 3D) DirectDraw surface. Added logic to render the schematic onto a bitmap that matches the scroll box control.

The TDxdControlBase class handles generic initialization, rendering and device context manipulation. TDxdBitmapControl creates a bitmap to cover the client area of a given window control, renders to it using the inherited logic, and blits it to the desktop. TDxdControl initializes the DirectDraw engine, creates a back buffer to use for rendering the schematic, then blits it to foreground.

Cleaner selection logic that avoids certain redundant serialization passes, and avoids repaint.


77 Sunday, April 06, 2003

GUI {

Trapped the AngleZ scroll bar to roll over from 0 to 359 degrees upon a LineUp event.

SelectNode loads/saves the angle values.

Fixes to creation and destruction of DxD frame buffers.

}

Caught a problem in the doubly recursive GetCrossMatrix function: the algorithm was climbing up the "from" and "to" trees beyond their common parent doing unnecessary and non-commutative matrix transformations backward and forward. Changed this logic with a recursive GetCrossMatrix2 called twice from GetCrossMatrix and limited in depth to a pre-calculated common parent. First it collects bottom-up the inverse matrix along the "to" branch, then it calculates top-down the straight matrix for the "from" branch.

The logic of serializing line segments was of extreme difficulty. A rethink was required to correct the numerous problems of coordinate spaces.

The algorithm needs to show wires and cables to cover the following cases:

Criterium 1: Wires of the following type:

  1. First level wires (wires between children of the current node).
  2. Self wires (wires to and from the same node).
  3. Secondary level wires (wires between grandchildren of the current node).
  4. Un-even wires (wires between nodes located at different depth level).

Criterium 2: Unbound and bound wires.

Criterium 3: Wires belonging to a grouped or to an ungrouped cable.

Criterium 5: Wires affected by bend-points

Criterium 6: Wires affected by rotations of the endpoints or of the parents of their endpoints.

Correcting the LINE::SerializeSegment function was a nightmare. It proves again that only the cleanest logic, one that is defendable in theory, can solve most difficult problems.

The bendpoints are now set in the coordinate space of the first node that covers the topmost cable. The LINE::MoveBy logic needed to reflect this by transforming the bend position into that line's parent coordinate space. Inside SerializeSegment a transformation to the coordinate space of the respective endpoint is done and undone around drawing of each segment. Recursing the SerializeSegment function for sub-wires is also done in the original coordinate space that was received from the parent node of the top cable.


78 Monday, April 07, 2003

GUI {

Added two scroll bars to the Schematic Editor and hidden those of the ScrollBox as they were troublesome. The paintbox is now the same size as the visible area of the scroll box. Changed the size/origin of all serialization for drawing and for pick correlation. Removed several conditions that lead to flicker. The new scrollbars reset on right-mouse-click.

}


79 Monday, April 14, 2003

GUI {

Enabled another visual flag for nodes: vaNewCable=0x200 that engages the core to expand a cable as a bus of sub-wires. Added a GUI option flag to configure NODE::GuiFlags.

Enhanced the TypeMaster Reporter to show the type sizes as well as the total sizes of instances (in bytes). We need this to help with our work on reducing the size of K5's types. Double-click refreshes the reporter view.

One important refining that speeds-up the screen refresh: on any paint message just blt the DirectDraw buffer onto the screen without building the scene.

Using the status bar to show us when and what is being hooked by the wiring tool. Pressing ESC anywhere will reset the wiring tool.

}

Fixed a problem with the wiring logic that was not preventing wiring of lines.

STREAM::Ellipse didn't need the len parameter. STREAM::PolyLine was a misnomer and was renamed to Polygon.

point::PointOnLine was renamed to point::IsOnLine. Added new point methods to support the new bus drawing algorithm:

Bisector (point A, B, C) calculates point D (this) as the foot of the bisector of angle ABC onto segment AC.

int IsInsidePoly (point* pl, int len, int& segment) is a hit-test function that breaks-up a polygon, which is described in terms of its consecutive vertices, into quadrilaters and hit-tests them, returning the count found.

The IsOnPoly hit-tests the perimeter of a polygon. Perpendicular (point A, point B, point& A2, point& B2) translates the A and B points into "this" coordinates, then reflects them against the first bisector to obtain A2 and B2 points which describe a perpendicular in "this" to AB. The IsInsideTriangle (point A, B, C) hit-tests "this" point using the "sum of angles = 360" theorem applied around the testing point with a tollerance of some 3 degrees. GetAngle (point that) calculates the angle in radians between vectors "this" and "that", and it is used by IsInsideTriangle. Enhanced LineAndCircle to also return the intersection points, as needed by the new cable drawing algorithm.

Added a new visual flag to nodes vaNewCable=0x200 that affects the line drawing algorithm in that, once set by GUI, it engages the new bus drawing logic.

GetWidth was a misnomer and was renamed to GetDepth which also calculates the recursive width of the bus.

New and difficult functions: SerializeCable, CollectBusInfo, SerializeBus working together to achieve a bus drawing. CollectBusInfo collects the perimeter vertices of an expanded bus that has bend-points. SerializeBus then allocates the width of the tube to sub-wires, and draws them. SerializeCable is like Serialize for lines.

Heavy changes to SerializeSegment to do the minimum necessary coordinate space transformations. Also, the logic discriminating vaInstance lines was removed as doubtful.


80 Thursday, April 17, 2003

GUI {

Added a group of controls specific to lines to configure the position of the left and right labels and the gap between the wires of a cable. Added a few checkboxes to configure LINE static options. Corrected some profiling calculation using quads instead of ints. Caught and corrected a bug that offset the device and viewport coordinates when drawing without DXD caching. This was due to DiagramHdc sometimes coming from Windows without the ADVANCED mode set.

}

Label drawing for lines has changed. The core is not using the nodal aspect of a line to draw labels. The aim was to free the nodal aspect of lines for other "uses". Each line now has a pair of label position parameters to draw labels at a certain position in stretched coordinates, at either endpoint. A label position parameter value of 101 (1% over the limit) renders the label invisible.

LINE::SerializeSegment and SerializeBus now calculate, use and return the length of the line.

LINE::GetMidpoint calculates the middle of the minimum length segment created between two node shapes intersecting their center line. The function has a difficult job to do and consider all the coordinate space transformations between nodes.

GetNonCableParent climbs up the parents of a line to find the topmost cable and its parent.

Added another data field to LINE, BendInfo.

Another point function to return the angle between two segments in degrees.

Changed the initialization order as wanted by the new auto-binding logic for recursive (self) relations.

The auto-bind mechanism has changed to establish cables even for recursive relations.

Heavy changes to LINE::SerializeSegment:

  1. The order of drawing line segments between bendpoints changed (the order of recursion) so that some global information (such as the length) may be collected about the line before drawing each segment. The core needs this to position the lables on the line parametrically.
  2. Not using the node aspect of the line as virtual bendpoint. This is to free the role of a line's node aspect.
  3. The new virtual bendpoint is calculated as the middle of the MinSegment between two node shapes.
  4. The function doesn't draw that residue of a relation over the top of the cable that contains it. (vertical relations)
  5. It draws two labels around each endpoint and in the direction of the line segment.
  6. The behavior of this function is configured from LINE's static flags which are managed by the GUI.

LINE::MoveBy was affected by the change in thinking about the virtual bendpoint of a line.

SerializeBus differs in behavior to SerializeSegment in that it considers both L&R sets of bendpoints as one, and descends them left-to-right rather than center-to-each-side. A series of implications here to do with picking, breaking, deleting and moving bendpoints on either the L or R buffers.

Many changes to the order and behavior of SerializeBus to allow us to pick the control line of the bus, pick the bendpoints, pick something from behind the bus through the spaces between the wires, etc.

SerializeCable doesn't have pick correlation anymore to allow the wires to be picked.

Caught and corrected a problem with coordinate system transformations between NODE::Set/GetPosition that didn't balance.


81 Saturday, April 19, 2003

GUI {

Logic to toggle a line flag to drive the new cable drawing logic and trigger a screen refresh without selecting. This is for testing purposes. Also changed the UpDown detail level to reselect the node, also for testing some aspects of the core.

MouseUp now uses the status bar to show correlation info for lines.

}

Another NODE visual flag vaExpanded, to drive the new bus drawing logic and the calculation of depth and width.

Caught and fixed a problem with the serializor crashing on the wrong virtual method when doing PointerArithmetics. This was due to the virtual inheritance class layout of TYPE. PointerArithmetic must be the 3rd vfunction in any class layout.

Reduced several unused parameters to serializer helper functions.

SerializeCable was merged with SerializeLine and SerializeBus was merged with SerializeSegment. A new function: SerializePartSegment, deals with one segment of the bus (between two bendpoints) and draws everything within recursively. The difficulty here was to make it look like SerializeSegment which is recursive from line's midpoint towards both ends. Heavy changes to map segment indexes between the two models: F2T and C2F+C2T. Made a helper function to just draw or select a wire segment.

A change to NODE::Serialize, prompted by an attempt to reinstate the former draw-by-reference serialization logic, to prevent serializing relations in the first pass, and allow them in the second.

Changes to segment index mapping inside MoveBy. Preventing collinear bends reduction in ReduceBends for generalized self-lines (lines from a child of the "to" node, or vice versa). This is a refinement on the "virtual bendpoint" concept, where, since there are no real bendpoints, picking and bending is more difficult if followed closely by the reduction algorithm.


82 Monday, April 21, 2003

GUI {

UpDown control to handle the gap between cable cores. Scrollbar to position the center wire label. Logic to wrap the values. Better hints for label and gap positions.

}

point::LineAndEllipse returns two explicit points of intersection. Corrected point::Perpendicular to do a rotation by 90 degrees.

The idea of a tube is gone. LINE::CollectBusInfo, SerializePartSegment, DrawOrSelectSegment, are gone and replaced by the new and powerful SerializeSegment(New) that recurses on two dimensions: longitudinal and traversal and draws parallel recursive buses. Split GetMinSegment away from GetMidpoint for uses in the new SerializeSegment routine. LINE::AbsorbMidpoint is obsolete now since the line's node aspect bares no consequence in bending, that role being undertaken by the "virtual midpoint". Corrections to GetDepth. Removed the co linearity test in reducing bendpoints.


83 Wednesday, April 23, 2003

Merged TQList into TREE. TREE::FindFlat and TREE::InsertBefore are merged from TQList. Needed to realign the Rose category for flat insertion. POSIX::TMicroPorocessor descends from TREE now. FSYS::TFileFilter is a TREE now. NODE_CLIENT is a TREE.

Fixed the LineAndEllipse intersection problem.


84 Tuesday, April 22, 2003

Fairly deep changes to the interface:

1. Changed the CoolBar for a ReBar control that doesn't flicker when individual toolbars are hidden/shown.

2. Made 2 new toolbars: the LineEditor, and StaticEditor. Static editor should hold all of Core's general behavior flags.

3. Distributed the tools to the various toolbars rather more logically. Worked on their hints.

4. ZOOM function added between -10 and 10.

5. Finally found the way to use the mouse wheel to scroll the schematic. (Use CTRL+Wheel to move sideways);

6. Added some intention of help support.

7. A change to allow keyboard input onto the Schematic.

8. Added a splitter to keep the toolbar of a user-chosen size.

A bit of cleanup of vFlags that have remained obsolete. Reversed AutobindEnabled and turned into a static line flag.

File format changed by serializing LINE::BendInfo.


85 Thursday, April 24, 2003

HELP{ Added the first context entry points to NODE, LINE, Relation, Thread.}

GUI {

Option flags to configure the Expert Reductor: Recurse, AbsorbInitDone, FactorizeMethodParameters, RebindMethodsIntoCable.

The first HELP system with some 10 linkes.

Resolved a crash condition when zooming..

}

I've been working for years to get her: symbolic reduction of code. RELATION and TYPE cooperate to do this with the following methods: int CompareTypes(L,R,l,r) that compares types L and R returning a code value and relations l and r set to the last point of identity; ReduceMethods, a function that iterates the relations of a type to find methods, then matches each method against the ancestor's looking for similarities that may be either completely or partly factorized; ReduceMethodWithScope recurses the process of reducing method prototypes up the inheritance line. New helper functions: GetLasRELATION and get_Client.

Performance improvement by a factor of 4, by passing through the Serialize (segment) the direct and the inverse coordinates transformation matrixes, as well as the serialization depth. The hierarchy of node endpoints doesn't have to be passed up and down any more.

Simplified NODE::Tile by using LINE::GetMidpoint. A tile operation will recursively reset the bend information.

Added simple logic to color the cable.

Reduced any bool to boolean.

Why is the HLP file getting corrupt?

Had to change the way methods are loaded from a Rose model: they insert into the Class of their declaration. This means that they cannot auto-bind. It's for the Expert Reductor algorithm that needs to find methods other than by searching through cable cores. Until we find a better way, all methods are loaded from a Rose model as virtual.

Promoted int, void, long, float, double, char, boolean, lstring, zstring to ABSTRACT.


86 Saturday, April 26, 2003

Sacrificed NODE::Detail to become ConstantType and keep the file format the same.

Realigned the abstract model. Node UserData renamed for Handle. This field is used by bitmap nodes, constants, streams canvases and memory streams. Merged the notion of a constant into NODE. Removed the label class.


87 Friday, May 02, 2003

Extended MORPH::CreateLine and CreateNode with a flags parameter (shift codes). CreateNode is using the shifts to create a proxy node, which is in fact a "USE" cable endpoint.

The new and quite sophisticated mechanism of USE lines. These are hollow tubes that connect a scope directly with another, one through which we "see" on the other side. The USE line has several properties:

  1. It wires from itself to another type.
  2. It auto constructs by grabbing all the sibling relations within the cable.
  3. When deleted, it restores the parenthood of its sub-cores.
  4. It allows a different size/shape/orientation than the target type.
  5. It simplifies/narrows the super-cable through which it runs.
  6. It is recursive on its "from" endpoint.
  7. A USE cable serializes by serializing all of its cores regardless of their parenthood, in the coordinate space of the parent of the "USE" relation.
  8. The label of a USE cable is built by AggregateRelativePath.
  9. They do not AutoRemove, and their cores do not AutoBind.

NODE::IsUse is used to test for a "USE" cable.

Enhanced the cable serializer with the "pseudonode" parameter that may either be the actual node or its proxy USE. Changed the cable serializor to draw arrows only when the line's direction can be determined sufficiently. If the sub-wires are to/from the same node as the cable, then the arrows are drawn inside, rather than outside the cable.

The Interlock is now a TREE and was promoted from LINE to NODE. Therefore, NODE needs a Set/UnsetTypeMaster.

Changes to the iteration in NODE::Serialize, as there are new cases.

BendPoints is now dynamically alocated BEND_BLOCK points at a time. Serializing BendPoints has a problem.

Simplified the model. lstring and zsring are gone.

Cables require a line to insert within the cable's order, which is different to the scope order. Another tree aspect, Order, was required for lines. AutoBind doesn't alter the direct parent. It manages the Order structure instead. Node constructors need to be called with proper parent pointers now. AutoBind will create a cable for any two or more relations to and from the same place.

LINE::FindCreate was extended with a count parameter, and a "created" return flag.

NODE::Tile was corrected and then enhanced to take into account the strength of the node's use, the count of lines in and out of it. The more connections a node has, the larger a perimeter the node must have.

The backward compatibility of the file system was lost from V6 to V7. Too many changes in the morphing cross-reference.

Found and fixed a problem in HitTestPoly that was skipping vertices, resulting in errant testing of a triangle shape.

Turned the internal name buffer of SerializeExport to static (unsafe for threading) to avoid stack overflow.

A fix to ReduceMethods.

Inserting a tree into null is equivalent to removing it from its branch.

Added TREE::SelfTest


88 Monday, June 09, 2003

In short:

Single inheritance. Depth Limited Serialization, Visible Interface Serialization. OBJECT-TYPE TypeInfo Interlock. STREE. OBJECT-STREAM interlock. Proxies. LINE not NODE. Selectors and Constants. NODES instances of Relations. Lines instances of operators. Recursive cables.STREE Line femurs

Reasons for change:

1. The greatest problem is the stability of the design. I must reach the kernel form of the model. On this depends the effort of documenting.

2. Another major problem is always the stability of the binary files. If the format keeps changing, I cannot build decent models as I know they won't survive the next version. This is going to be an ongoing concern, but it is made a lot worse by the unsettled issues present in the design itself, which lead to severe inversions of the file format.

3. Refining and reuse of principles that prompt to reduce the model.

4. Questions have been answered, and the design needed to reflect that.

5. This version is probably the most severe break (and breakthrough) in the history of the system given the multitude of interdependent changes that took place. In other words: given the system's current degree of coherence, to exert a semantic change means to rework it all. Given my principle of never archiving broken work, it follows that the system has changed everywhere for this release, placing the burden of justifying onto this particular report entry.

6. The morphism Core has vacillated from TYPE<>Relations, TYPE=RELATION, back to TYPE<>Relation, and now to Relation>Type. A review of the original reasons follows, as extracts from my log:

a. "Sunday, September 15, 2002 The heaviest changes since Rev 1.20 of the ABSTRACT in 21 Feb 1999 17:34:08 Alexander_Mihail_ALFA ( The semantics of the language have changed. Functions and relations are now classes): Rationale: Lines are instances of Relations. User relations are also relations. Deleting a user relation deletes all wires of that relation. Changing the attributes of a relation affects all wires/lines of that relation. Relations are now both Lines and Hierarchies by double descent. The major problem of unique instance identity for lines needed to be solved. A relation can only belong to one instance chain of a Hierarchy. It follows that a relation cannot have two NODE ancestors as it would directly inherit from LINE and from HIERARCHY. Therefore LINE and Hierarchy had to descend from NODE virtually so that Relation only ever gets one NODE aspect for visuals and for chaining instances.

b. "Monday, November 13, 2000 Two major changes have taken place. TYPE and RELATION are now different classes. This is how the Morphing Core begun anyway. I was not convinced that types and relations can be one and the same concept. I am still uncertain either way, but right now, relations are many and to handle them as type masters seems prohibitive. RELATION is a LINE descendent and TYPE is a NODE."

c. "21 Feb 1999 17:45:32 The semantics of the language have changed. Functions and relations are now classes."

d. The Morphism Core was making a distinction between Type, Relation and Method before 21 Feb 1999

7. Beside the problems incurred in the normal development of the Core, the TYPE-Relation problem was fundamental. Between d and c I've realized, despite everything that a whole world believes, that there cannot be a distinction between a Type and a Method, and the two concepts merged, enticing me to also merge them with the notion of a relation. At b. (13/11/200) it was more a matter of efficiency that split back the Relation away from Type. It seemed prohibitive to have relations (then coded as lines) also maintain a Type aspect. It wasn't clear back then that it would actually be necessary. Although the 15/09/02 revision of the core made no difference between relations and operators, and considered that both generated Lines, and that Selectors were Lines linking a "from" relation and a "to" relation, it recognized that a Relation needs still to be a Type, and thus track its generated instances. A Relation in the previous version was a double construct: Type and Line, but with a merged Node aspect (all carefully arranged through virtual inheritance. The fall of the latest approach was prompted by the seemingly benign inconvenience of Lines being Node descendents without good reason.

8. This version considers that Lines are not nodes, although they all are TREEs. A Relation is a Type but it is no longer a Line. Relation contains a Line instead. As Types and therefore as Nodes, Relations insert into their source Type. Here their Node aspect plays a role in drawing the Type's fields, which has been a long standing goal. The Type aspect of a relation comes into play to generate selectors like before, only this time, Selectors are just Nodes an not Lines. The Connector/Line contained aspect of Relation is needed for all the usual graphical reasons, but also for finding the aggregation endpoints. A new distinction emerged as soon as Relations began to generate Nodes: who generates Lines? The answer came, not without pain, Operators. Operators are now special Relations that generate Lines instead of Nodes, which is much better formalism.

9. A new chapter in the "Morphology" section of the "Concept Report" goes into the details and the reasons behind {OBJECT, TYPE, RELATION, NODE, CONNECTOR, PROCESS}.

10. New thoughts went into the redesign of TypeInfo based on a common STREE structure:

a. The notion of Type Pivots,

b. the concept of a handle external to the system

c. the new formalism of Constants and of Selectors

11. A new scheme of Object interlock was devised based on the common STREE that opens the path to:

a. Implementing Real-time Processes;

b. network-wide serialization;

c. Concurrent serialization

12. OBJECT has now 3 STREE structures, so it became a fairly expensive structure of some 115 bytes and growing. This actually reduced the size of other descending structures. As a consequence, a TREE has 4 STREE aspects, and to manage a LINE's femur we only need one STREE. Major rework on Lines was required as a consequence to not use TREE for endpoint femur.

13. A new concept of "depth-limited-serialization" (DLS) has emerged and was implemented, with thorough rethink of Exports/Imports, of Missing Instance Creation, of serialization order, etc. The core can serialize in and out "sketches" of models and either reconstruct the dependents or bind to those found to match. This is a complex matter discussed in the Progress Report.

Specific changes:

1. Please refer to the Activity Log, Progress Report, or Concept Report for discussions on Types, Objects, Constants and handles.

2. Enhancements to SelfTest everywhere to catch most subtle errors.

3. Reverted to single inheritance. Simplified the VMT macros.

4. Rationalized all constructors.

5. Added the ISTYPE(Type,AncestorType) macro to generate the IsType functions, which are better and faster than navigating inheritance relations that may not have bound yet.

6. TYPE constructor simplified: it doesn't take the versionid, prev, next, children, max_size, info, and the comment string. It was seen from their well established use that these fields mean little.

7. New faces for bool, Operator, and some others. All bitmaps in the core are now 256 colors.

8. MEMORY_STREAM can grow or shrink now, reallocating its buffer space. The buffer is considered by the Kernel to be a handle to the outer world.

9. TCanvasStream implements a new SetFont function that takes a 64 bytes (LOGFONT) structure as argument. The new core inherits promotes and overrides fonts. Canvases are now capable of nesting, as required for recursive drawing of node faces and their type masters. The device context of a canvas stream is considered by the Core to be a handle to the outer world, as discussed in the White Papers. Corrections to WriteBitmap to support the changes to GetFace.

10. Reversing HylaFAX has prompted with the matter of methods with variable argument lists "...". The dot is currently reserved by the path builder RelativePath and by the FindCreate algorithms. So a name replace was needed when reversing for such methods.

11. MORPH::CreateLine has changed to reflect the new kernel type family. Now there are much fewer cases of wiring: op(REL-REL), rel(TYPE,TYPE), =(SEL,SEL), =(SEL,THREAD). The function is responsible for creating Assignment Operators, and for instantiating them.

12. A USE relation instead of the former USE line.

13. Logic to support the K5core front end in finding the Workspace and inserting nodes loaded from the various tests.

14. Organized the class IDs better onto ranges. Compacted the gaps.

15. Fixed a crash condition in point::distance. Resolved a bug in point::SegmentAndSegment.

16. Stack consumption:

  1. Reduced the serializer's stack consumption by separating Canvas serializer (which uses matrixes and point arrays) from the binary serializer.
  2. A def file for Navigator just in case the current 16M stack size isn't enough.

17. Changes to serializer:

1. Serialize doesn't carry the version parameter anymore. The version is kept as a field of any Stream. So only the system may have a version, not the individual types that comprise it.

2. TGrammar alias for TREE was removed from the model during cleanup.

3. Stream options so far: ofDebugEcho to dump any ReadBuf to console, ofDebugClassID to show (stream position:classid) pairs whenever encountered (important debug tool for mismatched load/store), ofDebugEchoText to dump onto console any text read. Bells and backspaces are substituted with dots when dumping debug data.

4. STREAM::Signature either writes the EnterpriseFileSignature field out to stream or attempts to match it.

5. Using OBJECT::Interlock instead of the old stream selection mechanism of markers. Finding a serialized object makes use of the generic STREE::GetFlatID, FindFlatID.

6. GuiAttach is now to do with Selector NODES, not with Relations.

7. Stream has methods for calculating the stack consumption of a serialization process. It calls the abstract OBJECT::Progress to display it.

8. Removed stream's cascade to children read/write logic. I forgot what that was for, but caused great trouble.

9. The file handle in the case of system files is considered to be external handles like the device context handle of a Canvas Stream, or the memory buffer of a Memory Stream, as discussed in the White Papers.

10. The new core inherits promotes and overrides fonts as 64byte buffers of raw data.

11. SerializeExport is primarily called by STREE::Serialize. This is a very delicate algorithm that looks for referenced objects that fall outside of the "umbrella" or are below the specified depth of serialization and recursively serializes offsets and class ids of the referenced interface. The mechanism will be known as "Visible Interface Serialization" (VIS). Depth culling is called "depth-limited-serialization" (DLS) for exports and imports. These matters far exceed the scope of this diff report and are discussed in the White Papers.

18. STREE is new. This is a non-object structure with Parent, Children, Next, Prev, FlatCount fields. It has a pointer to the Object that aggregates the structure. STREE unifies the following matters: TypeInfo, Interlock, Tree, Proxies, Line-endpoint. Access to its fields must be done through the procedural interface to set and retrieve objects. Its behavior is in part expected, but has several aspects that are quite complex. STREE should be the object of a separate discussion. It must be said though that STREE is the central piece of several mechanisms such as:

    1. the serialization process.
    2. the SelfTest
    3. finding by ID.

19. SINTERLOCK is just another STREE with an extra Data field to store extra pick correlation information.

20. Changes to OBJECT and TYPE binding

1. Objects and their Types relate through their TypeInfo structure by insertion of the first into the latter. Only one such parenthood is necessary within the same application of the Morphism Core. The TypeInfo.Children field of instances are considered to be handles to the outer world. Please se the "Formalism" and "Morphology" sections of the "Concept Report" for details on the recursive nature of the whole core.

2. Simplified construction/destruction.

3. Object containment is now reflected by the TypeInfo.Object pointer

4. Proxy interfaces have superseded and generalized the former NODE_CLIENT class. Any object in the system may have any number of "Proxies". Those are fake objects chained to the instance that follow the fait of the "server" object. Proxies are installed by the user software with virtual method tables that match that of the main object. Core activity on the server's methods involves the proxy methods also. The net result is that all proxies are "informed" of methods being called on the server, as if they were called themselves. The core uses the UPDATE_PROXIES macro on a selected number of functions that we need right now, such as name changes, tree insertion and step. Proxies initialize after construction without TypeInfo and Interlock, and insert their Proxy structures into the server.

5. OBJECT::IsExport is a generic interface to be overridden to calculate the visibility of a given type instance with respect to the serialization umbrella. See White Paper for details on the Export/Import mechanism, DLS, Morphology, and Folding Kernel Architecture.

6. The new object interlock mechanism based on STREE structures. Please refer to the corresponding revision log of STREAM for details on Interlock.

7. Serializing an object ends-up wanting to serialize that object's type master. If the type master resides under the umbrella it will also be serialized along with all its instances. (Heavy recursions here)

8. Progress function to support stack notifications from serializer processes.

9. The Morphism Core finishes its initialization by creating the USER type. The pointer is registered at the heart of the system in the new OBJECT::UserType field. Any algorithm may query this field to find when the core has finished initializing.

10. OBJECT::Free climbs up the containment chain to destroy the outermost complex.

11. SetTypeMaster is the default constructor/destructor. GetTypeMaster and get_NextInstance work by offsetting the TypeInfo structure.

12. OBJECT::DebugDump outputs a fully qualified ID string to the console.

21. Changes to TREE

1. TREE has an STREE to manage its direct tree aspect. Access to TREE's Next, Prev, Parent, Children must be done through the procedural interface that takes care of de-referencing STREEs to TREEs.

2. TREE also inherits another 3 STREE aspects from OBJECT for system integration.

3. Changed /simplified the constructor for TREE. It doesn't take a children pointer anymore. One will need to insert the children after construction. (Questionable decision, when considering large lists of children that may want to bind to their parent).

4. Visibility is the equivalent of a TREE's Options field.

5. FindCreate takes the "created" boolean by reference.

6. Another FindCreate method to complement the RelativePath algorithm. Together they implement the Export/Import mechanism of the Serializor.

7. A series of access functions such as CircularPrev, get_Next, get_First, get_Parent, get_Children, to cover all usage demands on a TREE type.

8. A TREE defers most of its activity to STREE.

9. TREES are constructed with Implementation visibility level.

10. RelativePath mangles Name#classId#instance along the path. The extension is mirrored by FindCreate. These two are used in the export/import mechanism.

11. TREE::IsExport calculates whether the tree is a branch of the umbrella or not. It also implements the first step of Depth Limited Serialization by returning the necessary comparison result.

22. LastData is a dummy integer that marks the end of BSS. Any address that is below this value is considered to be constant and will not be freed. The comparison is used in the new delete operator.

23. Changes to NODE

1. Using the new Interlock system for selection as discussed in the revision log (64) of OBJECT. and STREAM.

2. Nodes are now either Constants, Selectors or instances of descending types. Consequently the Shape field is to mean different things: the Shape of a constant describes the type of constant: C_DEC_NUMBER, C_HEX, etc. The Shape for a Selector is the type of runtime window associated. The shape of any descending type is the normal shSquare, shEllipse, etc face.

3. A new shArrow face to help with line endpoints. GetShapePoints is now static and accessible to Lines also to draw their endpoint shapes. I observed that the cases could be halved by symmetry applied to the bounds rectangle.

4. A new visual flag vaCover to determine whether the node face should be drawn over or under the node's children.

5. Reduced constructor. Nodes are tile able.

6. NODE_CLIENT superseded by the OBJECT::Proxies scheme discussed for OBJECT.

7. NODE::Serialize was split into Serialize, SerializeFace, SerializeChildren that work together to achieve the end result. SerializeFace uses the vaRecursiveView flag to nests the canvas and invoke the node's type master to do the drawing. Some nodes may be bitmaps as loaded by the core from resources. These have the shBitmapID shape.

8. SerializeConstant is a rework of the previous TConstant logic, and it either loads or stores a constant to the given stream in various formats. Constant data is serialized by the binary serializor. It took lots of fixing.

9. Several discrimination functions were needed: IsUseNode is needed to figure-out the special case of USE relationships that serialize differently to normal lines; false by default and overridden by Relation to return true if the USES flag is set. IsSelector and IsConstant return false if the node is in fact a descendent Type. Otherwise IsSelector and IsConstant return opposite results depending on whether the given node has a MaxSize or not. A node is deemed to be a constant if it has a MaxSize greater than 0, and consequently a non-void handle. A node is a Selector otherwise.

10. Selectors are instances of Relations, and they form selection chains along their main tree aspect. GetSelectorEndpoint returns the topmost parent of a selection chain that is not a selector, and is therefore a Type. Selectors use their handle value as indexes into their relation in accord with the relation's cardinal. GetSelectorOrigin returns a compound index into the related object.

11. handle_realloc is used to increase/decrease the size of constants.

12. Nodes have a Font pointer to a 64byte buffer of raw data. A node that doesn't have its own font inherits it from its Type Master. Fonts are serialized.

13. Nodes have now got static Options and Flags instead of VFlags, like any other Core Type.

14. Nodes have a type master pivot of type Relation.

15. Bits are now MaxSize, and MaxSize was promoted from TYPE.

16. Nodes inherit the OBJECT's interlocking mechanism now.

17. GetShapeNames returns a list of names out of a choice of three such lists depending on whether the node is a type, a constant or a selector. These names will fill the GUI selection combo boxes.

18. GetCommonParent moved to TREE/STREE

19. Expand is good for nothing now.

20. Nodes deemed to be bitmaps destroy their handle when DONE. Constant nodes free their handle.

21. set_Name, set_Information, and Step call the proxies to update.

22. Simplified type checking.

23. Only visible nodes get involved in tiling operations. True by default, Relations that have not been flagged as Instance are hidden. Such relations hide their fields and only draw their connectors.

24. A fix to GetXRefDegree.

24. Changes to LINE

1. IsType.

2. LINE is now instance of (generated by) OPERATOR.

3. Options and Flags, like anyone else. Static line Options so far are to do with showing labels and with the behavior of AutoBind.. One important option, ofShowSubWires is to do with drawing sub-wires for the new recursive cables.

4. New logic to select an endpoint face from a small selection. It doesn't seem that easy to conceive line endpoint shapes.

5. SerializeLine is new and meant to spare some stack space during binary serialization.

6. Lines may now involve their Relation Master to do the drawing, so GetDepth also needed to pass through the line's spacing. GetDepth now ignores the (stolen) sub-wires of USE relations to return a narrower width.

7. GetEndShapeNames returns symbolic names to fill any GUI combo box.

8. Discrimination functions: IsWire looks to see if the line is bound or not to a container. IsRelation wants to check if the line belongs to (is bound by) a Relation or to an OPERATOR. IsUse-checks to see if the line is bound to a container (!IsWire()), if the binding object is a Relation, and if the relation is USES. IsWire

9. Most importantly perhaps is that the LINE's endpoints are low level STREE structures instead of TREEs.

10. Explicit label position and center parametric fields. Explicit WireSpacing.

11. Threads needed to override IsSelector to return false.

12. Changed/corrected the circulators.

13. Evaluation logic must change and it is currently broken.

14. One major change, and a subtle one is that wires between two selectors are drawn to and from their field generators leaving them (selectors) GUI free. This has been an aim of the design.

15. Lines are, like I said, instances of OPERATORs Operators have wires in their content, to build structured assignments (the matter of multi-pin plugs). A wire instance of a structured operator may now expand to sub-wires by recursively invoking its type master. This too has been a long standing design goal.

16. Better logic to draw off-center line labels along the line at a certain parameter between 0-100%, and have the label draw upside down for any parameter between 100-200%.

17. AutoBind creates assignment operators automatically to wrap wires.

18. Line contributes to the serialization scheme by testing visibility conditions such as all endpoints outside the umbrella or not, and returning to STREAM::SerializeExport apropriate codes.

25. Changes to Type

1. Heavy changes to OBJECT. Details in rev 64 of ABSTRACT.

2. Single inheritance

3. Types have static Options and Flags like anyone else. Only one flag currently defined: flShowVisibility, which influences the workings of GetName to mangle type name and node name together.

4. CreateInstance now returns a NODE pointer. It has been reasoned that regardless of Type, the internal system structure that wraps the instance must be no less than (exactly a) Node. Nodes thus produced are Constants and have the Handle set to the instance data, which is the actual instance of the type. DeleteInstances is used by the destructor to clear the handle before destroying the NODE aspects. In other words: CreateType returns a minimum of NODE headers with constants hanging of their handle.

5. Added another test to SelfTest to look for mismatches in instance counts.

6. IsAssignmentCompatible returns the latest OPERATOR between two types.

7. CompareReduce now belongs to Type. Loads of changes to realign ReduceMethods.

8. RecursionDepth is a field that determines the level of longitudinal serializer recursion. Serialization is a multi-dimensional recursive algorithm. Serializing (to canvas) a node with a recursive definition is drawn to the specified recursion depth that is considered to be at the same level by depth traversal. It's to resolve problems such as the recursive turbine. Relations recursively draw their supplier type in the context of their client type.

9. Corrections to TYPE::GetSize to close recursions induced by self relations. MaxSize no longer enters the equation. It was all a mistake, I was making a confusion between Instance and header. Please see the Morphology section of the report to understand the difference between a coherence structure and the handle to constants exported to the outer world. The size of a type is determined by its relations.

10. A type's minimum size is determined by its constant with a maximum size. This is the mechanism of basic data type formation: make an initially empty type and call it "long", get it to create two constant nodes, call them "ZERO" and "MAX_LONG", give these constants binary representations of 0 (which occupies 1 bit, and 0xFFFFFFFF that occupies 32 bits, and the type long morphed to 32 bit.

11. Whenever the size of a type changes morphism recursively propagates to all the relations that have it as supplier.

12. DescendsFrom: Type doesn't use any specyally marked "bg" /BACKGROUND reltaions. Protecting against recursive inheritance which is allowed in principle.

13. Changes to circulators of a Type's relations.

14. User types are types created at the subsequent induction step (see "Morphology-Recursive/inductive reuse of the Morphism Core" section in the Concept Report. Such types get serialized by the core types by impersonation. We need to discuss this, because the language I'm using here is too loose.

15. SetFace is new and creates a node with shBitmapID and inserts it into the type. It's used by NODE::SerializeFace that may want to do a vaRecursiveView.

16. Types are neither constants, nor Selectors.

17. The Types instantiated by Type will subsequently instantiate Nodes.

18. Void, Bool, Char, Int built-in type masters are set to instantiate Nodes.

19. Most type masters are instantiated by Type, like before. LINE is generated by ABSTRACT.Operator, NODE is generated by ABSTRACT.Relation.

20. Generating constants for Int, Bool and Char to establish their sizes.

26. Changes to Relation

1. Relation is now a Type specialist. It aggregates a LINE by value. Single inheritance

2. Has Flags and static Options like everyone else.

3. Simplified the constructor.

4. Needed a set_To function that sets the Colors, Size and Shape of the Relation's field to that of the To node. This way on the Schematic Editor the instance relation fields look like their target types.

5. Responds positively to NODE::IsRelation() calls, responds to IsVisible depending on the state of Instance flag,

6. USE relation steal all the lines in the same direction. USE Relations restore their children wires to their initial "un-stolen" state when DONE.

7. More SelfTests for relations.

8. Corrected the circulators.

9. SerializeLabel is obsolete and removed. SerializeFace only responds to Instance relations, (relations with visible fields) by recursively drawing the supplier's face, and calling the inherited serializor.

27. OPERATOR is new. It specializes RELATION to instantiate LINES. Operator instantiates a LINE header with its handle set to raw data of MaxSize.

Gui:

  1. One of the recent aims of this development was to release the selectors from the Schematic Editor for a possible use into the Debugger. It has just come to that. TRuntime stream goes and collects all selectors and binds runtime windows to them.
  2. NODE_CLIENT is gone and replaced with the new OBJECT::Proxy mechanisms. TIsNodeClient descends from NODE now and binds itself to the target node as a proxy node.
  3. Selection is now based on the new Interlocking scheme of the Core. SSelectionInfo was demoted from core to GUI. See Progress Report or Concept Report for details on the new OBJECT interlocking mechanism.
  4. Introduced Cut/Copy/Paste using a MEMORY_STREAM. Depth Limited Serialization on both binary files and clipboard files. See Concept Report for details on the new serializer features. The memory stream is inserted into the Enterprise and it appears as a node into the Scope Viewer.
  5. Reading/Writing the file signature is now handled by the Core's STREAM Type.
  6. ZeroNodeSize was promoted to Core as NODE::ResetBounds.
  7. Toolbar decorated with icons from Core's resources. Each editor has the icon of the type master whose specifics are edited.
  8. GUI configures a Times New Roman font for the core's main Type Master.
  9. ScopeViewExpanding doesn't call NODE::Expand anymore. Expanding actually doesn't mean much in Core terms.
  10. Protections in the way of type checking on the selection. Most often dual aspects (and different pointers) are edited at a time. GUI response methods need to discriminate.
  11. Calling Reload dumps the content of all editors and reloads them.
  12. Better screen refresh after certain editing operations.
  13. A fix to mouse (re)correlation in zoomed conditions.
  14. Nicer bitmaps for the Debugger tool buttons.
  15. Font selection logic for NODES, LINES, and their descendents.
  16. Nicer bitmap for the color selector in the NODE editor.
  17. Changes to the toolbar for editing the object selected on the Schematic Editor.
    1. Global options editor (M) Click=tile, RClick=ResetBounds. Flags for labels, text direction and recursive wires.
    2. Node editor (N) Click=toggle between Selector and Constant. Shape has 3 contexts depending on what's selected: constant, selector, or type. The string names are fetched from Core. Palette selector. Size viewer.
    3. Tree editor: (T) Click=set font (dialogue), RClick clears the font and inherits.
    4. Relation editor (R): flags, cardinal, no click.
    5. Line editor (---): Left, Right, Center label parametric wrapping positions. RClick to go to the middle (100% of 200%) SubwireGap. Combos for endpoint styles.
    6. Type editor (H): Click to create instance, RClick to delete all instances of the selected type. Flags: ShowVisibility. UpDown for recursion depth.
  18. SelectNode has changed to invoke LoadEditors or SaveEditors on a node with a pass count. These functions may recurse once to load/store two aspects of the selected node in one go. Selecting a relation would load most editors (Line and Type). A relation may be selected in one of two ways: either grabbing it by its line, or by its field. Constants are edited through Memory Streams passed to NODE::SerializeConstant.
  19. We seem to have settled the design enough to say that one either selects a Type, a Relation, a Wire or a Selector/Constant.
  20. Label positions wrap at 100% and turn around backwards and upside down.
  21. Stream Options (debug) are brought to the interface in the Options tab.
  22. The core uses OBJECT::Progress to announce the stack consumption of serialization processes. GUI routes this function to a Gouge control.

K5CORE.EXE

  1. Changes to manipulate the current node
  2. This program must become a command prompt, first of all with file-system like navigation commands.

89 Tuesday, July 08, 2003

Every time I take a break from it, I start thinking like I used to before the previous session. Modulo 2.

Experiments with CONNECTOR templates.

The GUI canvas is now persistent. Changes were needed to TCanvasStream to configure the device.

STREE::RemoveChildren may now recursively unbind all items.

Using the Interlock mechanism to do depth selection. GetLastLock is new.

Fill flag for lines.

A bug fix to do with spacing-out and drawing a bus of wires.

Fixed the selection of the bus when a subwire is actually correlated. Using the depth interlocking scheme for selecting wires.

GUI {

A few methods for SSelectionInfo to help with depth interlocking correlation.

GUI can now set flags for LINE. The only one used so far is Fill.

Changes to unify drawing and pick-correlation. Persistent CanvasStream so that selections may be kept.

Fixes to Shape, Angle and Colour editor for the Connector of Relations.

Unification of code in the threaded loader.

}


90 Friday, July 11, 2003

Cleaned up the unused stream filter constants. A protection to degenerate angles in the point class. A bug fix for object delete. It must dismember the Interlock without destroying the children. A bug fix to do with destroying selectors with a non-zero index. Added the STREE::Promote function that takes a tree's children and links them to the parent. Should really be called "Emancipate".

Multiple selection support. The principle is as follows:

  1. The selection is persistent through paint/write serialization operations.
  2. Multiple Selection looks like a tree, or like a glove. The finger tips are the items ultimately selected. All other joints are needed for coherence. More-over, certain operations may be conceived over the whole glove, not just the fingertips.
  3. Pick-correlation follows the existing "glove fingers" of the multiple selection, one finger at the time, and changes its tip if necessary. The former pick algorithm used to clear the selection early-on to be sure it has a clean start. This one keeps the selection as much as possible, until it becomes certain that a node or branch cannot be part of the selection "glove".
  4. Picking a fingertip twice will unpick it. This mechanism is necessary so that the user may unselect something without having to discard a whole glove of selection.
  5. The new pick-correlator follows the scope from top to bottom and interlocks all nodes that hit-test positive (thus forming a "finger").
  6. Pick correlation is now cumulative. This means that if A->B->C and A->B->D are possible paths, and if A and B were picked as they hit-tested positive, picking again will also add C to the path if the hotspot still keeps within A and B and B is now also hit-testing positive. This mechanism was crucial in recursively correlating selectors through "dial" frames.
  7. A consequence of the cumulative properties of the pick-correlator is that while the whole finger is selected in one go, un-picking must be one level at a time.
  8. Lines are more troublesome for cumulative picking. Selecting a sub-wire means to select the cable if it was picked from a cable expansion, but it also means to select it alone if it was picked from the lose ends outside the cable. The decision to remove a wire from the glove is taken at the deepest end of a traversal recursion through the bus drawing algorithm, if the wire is found locked and of the same orientation as currently serializing. Picked wires interlock with their cables.

Selectors are hidden unless the ofShowSelectors is set.

SerializeSelector and SerializeSelectionFrame are new functions to implement the new selector pick correlation scheme.

NODE::SerializeSelectionFrame is used by Relation to draw a "dial" control around the frame of the relation's field. The dial control is a set markers around the frame that may be picked. There will be as many such markers as indicated by the relation's cardinal. The function is passed an index value to say which of the pick points is focused. If the stream direction is READ, the index is correlated and returned to the calling Relation. The dial also draws the numeric labels for each pick point as well as the full name for the one focused. This function is called by a selected relation to allow us to pick an index into the field array. Dials are expensive to serialize (imagine an array of 10,000 wanting to show labels along different directions), so they are currently shown only for the pick-correlated relation. SerializeSelector is also called by Relation and is used to tunnel into the "to" endpoint of the relation. It recurses the serialization process. A field face represents a discontinuity in the top-down organization of a morphing structure. They can "throw" anywhere, including onto itself. Relations are symbolic links from a type to another type. the target type may be locked by another serializor or by the current serializor in its previous step. Attempting to lock elements of the referenced type will be denied. This is a normal phenomena. If the two serializing streams cooperate, they could interrogate each other to avoid the deadlock/corruption. One of the streams agrees to interlock not the actual relation (the discussion only pertains to selecting relations), but a clone of it, which is none other then the Selector. It follows that Selectors are justified by the Interlock. Selectors can only be taken from public relations. Private relations are not shared and they may only be locked directly by a top-down descending serializor process (not in traversal/reference mode). The serializor constructs nested selectors children of each other, as if to say that the symbolic link was a sub-tree mounted at the link point. It follows that a field has more than one selector per index, as there may be more than one way of getting from root to that field. We have one of two reasons to pick a relation: we want the actual relation (to configure it, delete it, wire it, etc.) or to de-reference it to its subtypes. The selector index dialer works by looking-up all the instances of the relation (we now know that selectors are instances of relations, we worked hard for that) to find one that matches the index value as well as the Interlocking context. Picking the dialer without picking a control point will select the relation. A morphing structure is a grammar. Please see Concept Report for a detailed discussion on these matters.


91 Tuesday, July 15, 2003

White Paper Words:

This new dial selection mechanism seems very important and the key of the project. What does it do: it applies the principle of "folding" to exponentially reduce the detail of selectable objects. Imagine what a nightmare it would be to pick inside a symbolic link between two cities with all kinds of structured cables, pipes, highways, etc. So dialers are the solution. Another issue of crucial importance that the dialer has helped bring to a close is: what are these Selectors really? The answer is: "letters of the alphabet". No, I'm not crazy. The Morphism Core is a grammar. Its topmost formalism is G: [M, T, R, K] where T is the set of Types, R is the set of Relations, K is the set of keywords, and M is the Morphism scope or the system's Goal. The productions of the Morphing Core are of the following form: T::=kR, which sheds new light on what Types really are, as well as explain the notion of multilateral relations. So a type T is described in terms of its relations as follows:

(1) T::=k1R1; T::=k2R2; ... T::=knRn which is the equivalent of the following structure description in C:

struct T

{

T1 k1; // Relation R1

T2 k2; // Relation R2

....

Tn kn; // Relation Rn

}

However, usual compilers do not see structures like we do here, and to them the effort of analysis falls outside and around the structure definition, whereas ours falls within. To the Morphism Core the grammar is that which the structure describes, not that in which the structure is written. This is an important point of difference.

The grammar above is type 3 and generates a regular language. Since this was a limitation in our plans of generalizing the algorithm to undertake ALL functions of the Morphing operating system, we were prompted to rethink the concept of a relation as follows. Binary relations are the common ones everyone is used with. Multilateral relations are of three types:

  1. One supplier and multiple clients,
  2. One client multiple suppliers,
  3. Mixed

Relations with one supplier and multiple clients are also used everywhere mostly intuitively rather than formally. Any two or more types that relate to the same type are bound to that type in a multilateral relation of type a. Relations with multiple suppliers are perhaps harder to visualize. Example:

struct T

{

{T1; T2[10]; ...; Tn*} k; // Relation with one client and multiple suppliers

}

The mixed multilateral relation looks like this:

.....

{L1 kl1; L2 kl2; ...;Ln kln} {R1 k1; R2 k2; ...; Rm km}; //Generic multilateral relation. between types Li and types Rj.

....

If it seemed easy enough to declare then the hardest part is to use such declarations, which brings us back to the matter of selecting. Lacking syntax and semantics, most programming languages will use indirect forms to achieve equivalence. In a structural language such as C, C++, Pascal, etc. multilateral relations which is what programming is really all about are described through proxy types:

Example:

struct 3DPoint { int x; int y, real z, int colour[3]};

struct ZCube { point A, point B;}; // using proxy type 3DPoint

struct ZLine { point A, real B;}; // using proxy type 3DPoint

which could have been expressed as

(5) {int x,y; real z, int colour[3]} {ZCube A,B; ZLine A,B}; // A multilateral relation between two supplier types and two client types.

Multilateral relations are a higher form of control of a program's semantics and a greater level of reuse.

The grammar of the Morphism Core with multilateral relations are of type 1, which are language dependent and closest to natural languages. Okay, so the language of the Morphing Core is context dependent and the productions look like (5). Productions of type (1) are a subset of (5).

The greatest problem that any system (and hence a compiler also) needs to address is to lay multiple paths over a structure. The action is called selection. Selection may become recursive if the underlying structure is recursive. Modern programs employ recursive definitions for their underlying structure.

Every production/relation has a key feature by means of which it can be recognized and selected. In a text language the key is the actual name, in a schematic it's the position/size of the relation. A derivation through the grammar (a path) is a sequence of user pick correlation actions.

--------------

The latest dial selection mechanism in its current implementation has achieved to lead the pick-correlator to lay nested selectors over recursive structures. Another aim of the development was reached with auto-creation of selectors by pick correlating.

Selectors are totally gone from the schematic so they have escaped beyond the induction step and into the user space, which has been a goal of the re-engineering effort since Friday, May 02, 2003.

The new serializor is now consistent with the C++ Compiler module that can now reduce against the Core. To "compile" is to "pick".

How does it work:

The selection glove is preserved through pick-draw-pick cycles, and modified atomically only when it becomes certain that a new relation is selected or released. When a traversal is required (by evaluating another production rule), to draw the typeface of a relation or pick a sub relation of the relation's endpoint type, the dialer gets involved. The dialer may evaluate to either the relation itself, an index of the relation, or to nothing. If the dialer has selected an index of the relation then a selector is created. The new selector gets linked into the umbrella. The umbrella represents a controlled branch of the system's symbols table. Linking the selectors to the umbrella and passing subsets of it down to traversals was the key to achieving these recursive paths. The context in which the dialer works is reach in conditions. A discrimination variable is built in Relation::SerializeFace and switched afterwards. The variable is like a recollection of what needs to be done before actually doing it. A new stream direction flag was added to mark the stream when in traversal mode.


92 Sunday, July 20, 2003

Heavy changes to GUI to support multiple selections{

Changed the Scope View to allow browsing olong any STREE dimension: Scope, TypeInfo, Interlock, From and To. Added a bit-field and the necessary response methods to TIsNodeClient and ScopeView to configure the direction of travel on a per proxy basis. One may now expand by Scope down to a certain depth, then by "From"/Suppliers, then by instances, and possibly back again. This new mechanism has superseded the Reference View.

Reorganized the left-hand panels: no more tabs for the browser. No more filter tab as I have new plans for this algorithm, to serve it from the serializor. A toolbar for the browser instead of the pop-up menu. Added a button to sort the tree by names, as needed with large Enterprises.

Added a simple OPERATOR editor with just one flag that isn't used. That's for completion reasons.

Cleaned-up any unused or half used GUI methods. Consolidated user interaction through Actions.

Simplified GetSelectedNode to not think of the binary phase anymore in expectation of multiple selections.

Gave up the idea of SSelectionInfo::GetFirst in favor of that of glove fingertips.

TIsNodeClient::Lock is a proxy notification issued by the core's interlock mechanism. The method looks to see if a proxy is already available and destroys it before creating a new one. This is to match the tree order of the core's interlock tree.

The main canvas is now named after the current user.

The Expand mechanism is now multi dimensional.

Setting and clearing of fonts, moving and resizing are now multi-select sensitive operations.

}

STREE::get_Newest and get_NextAbsolute are two methods for iterating the frontier of an STREE. It was asked for by the multiple selection algorithm. A virtual OBJECT::Unlock was required instead of Interlock.Remove simply to match the Lock method and notify all proxies that the Interlock STREE has changed. A bugfix to do with flShowVisibility and TYPE::GetName.


93 Thursday, July 24, 2003

GUI {

Auto-expanding trees in the scope view where finally achieved by going straight to Windows to avoid recursion of the Expand callback. Bold tree node labels. SSelectionInfo and SelectionPhase have been superseded by the new PrevSelected, PrevFrontier and PrevFrontierLength fields. GUI doesn't care about interlock code and info values, once there is no need for re-correlation. New logic to compare two bit-fields to a mask and set/read the tri-state checkboxes of the type editors accordingly. This is necessary with multiple editing. LoadEditors and SaveEditors were turned to work in a loop of multiple selection. They now take the type of the edited object from the caller. This will be the more generic common type of the multiple selection. FactorizeEditors is a new function that gets called incrementally with the objects along the selection frontier (the glove's fingertips) and compares the object values against the current values of the editor. If mismatches are found the corresponding control is turned to a tri-state which depends on the type of the control. The more bizarre tri-state indicator I'm using is the carret set to Drag for editors. We now have an auto-expanding selection tree. The pick glove is never null now, and has the Canvas at the top. No need to set a font for MORPH, Core will use the Canvas font, which can be changed easier. Found the trick to keep the rebar shrunk. GetSelectedNode returns the newest interlocked object. Reload was trimmed down to almost nothing. A new MouseDown response for the color control to re-enable the FG or BG if they were tri-stated by the FactorizeEditors multiple select logic. The entire selection frontier gets echoed into the status bar, as built by the new OBJECT::GetPath function.

Summary of the new multiple-selection aware operations:

  1. Setting any flags.
  2. Editing of Name, NodeData, Relation Cardinal.
  3. Arrow endpoint selection.
  4. FaceChange.
  5. Rotation angles
  6. Line labels position
  7. Node repositioning and resizing.
  8. Line segments may be simultaneously bent to the same point. Line bends may be simultaneously dragged.
  9. Deleting nodes.
  10. Tiling.
  11. Setting FG or BG colours.
  12. Sub-wire gap for lines.
  13. Recursion level of types.
  14. Font setting/resetting for the selection

TypeMaster button now creates constants instead of selectors, and the Node button logic to switch between selector and constant was disabled. Selectors cannot be serialized to canvas, and along with the automatic selector generation logic, it makes no sense to turn a constant into a selector.

The new SelectNode logic mediates the interests of the last caller as specified with the save/load/multi arguments, and those of the nodes previously gathered in the selection glove. Regardless of what arguments say, no save is required or desired when going from one node to multi. Conversely, multiple save is wanted when going from multi to single. Normal operation for single to single, as instructed by the save/load arguments. This is a delicate algorithm. Although I can think of nothing else around here that isn't.

}

Multiple selection needed to find the common type of the selection frontier. OBJECT::GetFrontier was implemented for this reason. It returns two vectors (or the length of those vectors) with the linear sub-tree, and another with the fringe of the tree. OBJECT::GetPath constructs a braced text view of the sub-tree. OBJECT::GetInterType is probably a transient method that aims at comparing the fringe of the Interlock and return the common type. Things are not as easy here since the client code may look for either the external type or for the internal type of the fringe nodes. There is a difference between the type returned by GetTypeMaster and the result of the IsType call. In other words, an object may be of a user type, but understood internally as say a NODE type. The difference is that between Header and Handle. Please see discussions in Concept Report. OBJECT::GetClassType returns the internal (header) type of an object.

Changed the UPDATE_PROXIES logic to cope with the possible changes in the proxy chains while iterating them. This scenario was entered with the new auto-expanding pick-selection tree that GUI now maintains. IT works-out that different functions want to update the proxies at different places in their body. We should generalize proxy update behind the ENTER and LEAVE statements.

Corrections to the static virtual OBJECT::CreateNewInstance. The function is not finished. It needs to construct all the aspects of the interface of a solid structured object, at least along the direct path. This is a highly sophisticated matter similar to VIS.

OBJECT::Unlock takes a recursive flag. It also updates the proxies, as does Lock. All interlocking needs to go through these two functions rather than Insert/Remove to update the new auto-expanding GUI multiple selection tree.

Multiple pick-selection for lines. Multiple unpick for lines isn't working yet.

Lines coming out of cables pick correlate as interlock children of their cable's end nodes. Lines pick correlate as interlock children of their cables otherwise.

Serializer now takes the default font from the stream itself. Any node that was hit was drawn twice. Reduced.

Not using anymore the re-correlation return value from LINE::MoveBy, the code/info is updated locally instead.

Added code and info references to LINE::Serialize to do the style same hit testing as for nodes. Not yet implemented.

TYPE::GetAncestor and GetCommonAncestor are new and required by the new OBJECT::GetInterType style of logic.

Selectors are now created with the default vtWindow style, equivalent to shNone.


94 Saturday, August 09, 2003

Multiple fixes, and subtle changes.

GUI {

Building the frontier is very much different now. It's done in every function that wants to iterate. No need for global Frontier variables. The new interlock mechanism equipped with OBJECT::Change now informs the GUI on selection changes. Editors apply to the multiple selection by pressing the H button (ok). The Canvas Stream inserts into the root of Scope Browser. Load file on expanding a stream node, storing to file if dirty and children. GUI only allows deletion for non readonly nodes. The read/write streams insert at the root of the scope browser. Cut/copy/paste of multiple selections. Multiple fixes.

}

  1. Construction of the Morphing Core has changed as prompted by an attempt to serialize the MORPH Kernel itself. Creating a new instance of MORPH, ABSTRACT, STREAMS, etc is now possible. These are the Morphism Pivots discussed in the Concept Report. The problem is that once a new piece of memory is allocated for any instance that contains a Pivot, that memory is then initialized by its creator and bares the identity of the creator. The TypeMaster map is pointing to the original instance and not to the new one. The new mechanism that allows subsequent instances of a type that has pivots is called Cloning. The original type masters are called in to help only as far as set the function pointers (VMT). The collection of VMTs and methods forms the induction step of the morphing principle. Another construction problem that became apparent was with instances that are being type set when their container was already bound.
  2. SET_TYPE_MASTER takes instance, type master, and reference type as arguments. Cloning is resolved by chaining the instance to the given type master (which can be un-initialized), while setting the VMTs to that of the reference type. The pivot also gets initialized in terms of its VMT_NORMAL.
  3. Added a Serialize and a SetTypeMaster function to STREAMS.
  4. STREAM::Open and Close as needed by the new Stream nodes in the Scope Explorer. The system allows us now to instantiate and configure a stream node which would open when expanding and close when collapsed. At the same time the file content is read on Expand and written on collapse. We don't want this to happen all the time, so if the stream node has children and is collapsed then the file isn't read thus preserving the children. A stream node is only loaded from file if there are no children are currently present. To reload a stream one must first delete its children then expand. When collapsing a stream node the opposite happens: nothing if there are no children, or saving the children to file otherwise. A further refinement was needed here to allow a stream node to collapse and expand without touching the underlying file: the Dirty flag. This is a new flag for all NODEs. It can be manually set from the GUI like all other flags to indicate that the user wishes to overwrite. A new OBJECT::Change method branch was needed to collect information about changes to the xref. Every class supplies an enum with subsequent codes as parameter to Change. chName, chInfo, chTypeInfo, chTree, chPosition, etc. The method is called by any piece of the Core that wishes to inform the rest of the world about a modification that occurred. Change returns a boolean which is tested in order to abort a change if another part of the system disagrees when notified. The specialized types down the hierarchy may overload the Change method to drive the notifications and act on them. Ex: TREE overloads Change to filter out any codes that should be promoted up the tree hierarchy towards the root. Along this hierarchy sits STREAM::Change which could, but doesn't set the dirty bit. At present we want to avoid pop-up dialogues, so the dirty bit is manually set for streams. Another use of Change is to notify the changes to the Interlocking structure along a different tree hierarchy. This new mechanism has resolved the problem of detecting changes in the Selection Glove. There was a little price to be paid at the GUI end: we couldn't keep the former lead-in/LoadEditors-lead-out/SaveEditors scheme with multiple selections. The M button now serves as the Okay button for the entire editor set.
  5. As strange as this may sound, streams also serialize to other streams.
  6. Memory Streams don't open and close.
  7. Streams can auto insert to trees with the new constructor. No more children parameters for NODE initializer.
  8. STREAM::Direction is now the STREM::Flags as everyone else in the Core. Added the STREAM::Truncate method.
  9. Changes to FindCreate and to RelativePath algorithm to allow caller selectable separators for the path. For export/import paths I'm now using non printable characters 1,2,3. This was prompted by serialization of streams with filenames that have extensions.
  10. All classes now have a GetDirectAncestor method, which is a static virtual.
  11. Fixed a bug in CAT_COMPILE with MaxSize set to 0.
  12. Not setting the system structures for CAT_USER was one of the reasons for failing to serialize the Kernel's image. It also needed a Type pointer of its own, as well as the IsType, GetAncestor, etc.
  13. The new Autoload.BIN stream instance is always present in the Core. These will grow to mean "loadable modules" treated graphically like all else here. This is solid part of the MORPH structure. Added the HeapBuffer at the end of the kernel map to anticipate structure morphisms. Another heap is provided by the USER scope. CAT_USER is also a type pivot.
  14. Added another NODE flag to tag nodes as read-only. The Core constructs the entire map with this flag set. It is overridden from the NODE::ofReadonly flag.
  15. MORPH::CreateNode was enhanced to create instance with drag and drop for any type.
  16. A much needed IS_OBJECT_INSTANCE non virtual function to test any structure before calling any of its VMT. It is used by the SelfTest branch of methods so that they don't crash on bad objects. It works in conjunction with the IsObjectInstance virtual that no one must ever overload (needs to become private); This VMT entry is a signature for valid VMT and the test fails if (*(int**)this)[5]!=IsObjectInstance.
  17. Lock needed another method: Locks so that together may correctly inform the GUI proxy chains of Interlock changes.
  18. Lock takes the "multi" parameter to instruct it not to unlock the sibling interlock branches. This is a solution to managing the multiple selection glove. OBJECT::Unlock takes two arguments now, the older one to recurse and unlock everything down the hierarchy, and the new "promote" flag that tells the object to pass all the sub-locks to the parent. The IsLocked method has changed and it can discriminate between a given owner and any other lock owner. It works by recursively climbing-up the interlock tree to find the potential owner. This new approach resolves the mutual exclusion problems with streams contending over the same object pools. The core now draws two different highlights, one for the local canvas, another for any other canvas that may lock a selection. With this mechanism in place, the Core tracks multiple users working (selecting) in different views. Fixed a bug that failed to update the proxies when trees were destroyed, streams in particular.
  19. One special and difficult problem is that of locking relations and their dial. Even in "single" selection mode the dialer represents an exception: relation and selector are locket as siblings. A new selDialRelation was introduced as a special lock code that disables the interlocking branch reduction for those nodes. SerializeSelectionFrame moved to TRealtion as SerializeDial. The look of the dialer has changed so that less is drawn for unfocussed relations. Simplified the dial selection logic.
  20. NODE::Step was promoted to OBJECT level. It is called on important activity such as Insert/Remove of trees. It's final aim is to update visual daemons. I can see it being taken over by the much stronger Change channel.
  21. An attempt to resurrect the Insert(replace) logic. This needs a lot more thought now. The Concept Report discusses the matter of Merging and that of Mimetic Absorption. Plenty of thoughts, not yet implemented. It won't be easy.
  22. Changes to the serializor to allow relocation. I found that VIS was forcing the loading process to a context similar to that of storing, and at exactly the same depth with respect to the Root (MORPH). There were many scenarios of scopes that were too loosely coupled with the core to warrant VIS. Several new orders govern FindCreate now: leading path ascendents may be trimmed to fit, and thus the scope may load higher then stored; serializing an instance doesn't recurse to serializing its type master if the type master can be trivially rejected as outside of the umbrella; anybody that is locked but not by us is ignored from serialization; we're now serializing type instances as potential exports.
  23. Serializing is now further complicated by the new stream nodes that load on expand. These nodes only ever want to serialize their children, and not themselves. It prompted to formalize that the umbrella node may be deemed to be outside of the serialization cone, so the cone is open. An open cone doesn't contain its surface and vertex.
  24. One important point that we're beginning to make here is that THREAD is STREAM. More on that in the Concept Report. The Thread construct has been demoted from abstract as a Stream specialist.
  25. The Core doesn't need those bitmaps (they were an experiment).
  26. A great deal of work on picking lines. Interlocking them when picked from within the cabled area will highlight them and glue them to glove fingertips. Any dragging operation in this case will apply to the ultimate cable, though. Selecting the wire outside the cable interlocks to the cable's endpoint node. Logic to have it all strongly ordered and pick before the underlying node. A malfunction exception remains if a wire crosses an area where there are no nodes underneath then no selection finger to it may be established.
  27. Due to new and more solid views on what the morphism induction step means in generating a world, neither do we need to wire any background inheritance relations. Changes to the way types and relations calculate their size, we're using MaxSize again. A crash protection to GetMaxConstant. TYPE::GetAncestor is simpler.
  28. Rose model loader: improved logic that rotates type masters to speed-up the loading process.
  29. Cut and paste was corrected, and it is now multi-select capable.

95 Thursday, August 14, 2003

GUI {

Changed the principles of having multiple forms in the GUI. A hidden tray form starts first. Launch a Navigator from the local menu or by double clicking on the tray icon. The tray icon is now responsible for initializing all the image lists that are now shared by all forms. Creating new forms is much quicker now, as querying the type faces of the Morphism Core is only done once by the tray. More subtle progress icons with colored squares. Resolved the matter of showing the progress indicators as tree node overlays.

A major change: the navigator main form is launched on its own thread. The TUserThread is new and a descendent of TREE. It constructs a UserInterface in its Execute function.

Once the Navigator windows were to construct and destruct independently of the tray, the DirectX initialization needed to be done in the tray form, once before any other form that uses it and potentially closes. It seems that DirectX initializes upon the first call to the interface, and no matter how you release things, it remains initialized until process closes. Tray is now calling InitializeDxD and ReleaseDxD.

Changed the Enterprise Navigator's icon to the face of a nasty devil.

Enhanced the TIsNodeClient class to return a pointer to the associated GUI. The UiChain and DeleteNode are now obsolete. Is a When GUI receives a Change notification at the level of its CurrentNode then the schematic editor will need refreshing. That's done in a timer as driven by the new PleaseRefresh counter. Actual refresh is also driven by the state of the new WillDrawAnyway flag.

A whole set of protections are now needed to deal with the real-time tree updates.

The GUI is not unlocking the glove prior to loading or storing a structure in a file.

Loading a file is done on a loader thread that's now a specialized THREAD. These threads lock to the current thread upon construction, so there's no need to insert them into the Scope View.

}

The Step function was superseded by Change. Several new change notifications for node and line.

THREAD has become functional. New thread methods: GetCurrentThread, StartRoutine and Execute. Threads have a Current pointer which is allocated on local thread space TLS. Enhanced the Lock function to deal with thread interlocking. The Lock function takes another argument now that tells it whether the caller expects to block or return. Locking onto streams results in the current thread being locked. PleaseUnlock is obsolete. OBJECT::Pid is renamed to Semaphore so to avoid adding another field to OBJECT and preserve the file system layout. All objects are semaphores and they can claim each other, eventually locking onto the current thread. The current thread is locked onto the target before calling the OS_LockSemaphore function which may block. This is so that the tree of locks is properly bound. OBJECT::Lock is also looking to avoid lock cycles and therefore any deadlocks. Interlock change notifications cannot be sent along paths with cycles. GetFrontier, GetPath, Promote, Change now implement protections against cycles. There are cases of constructions without a type master, when a GUI instantiates a descendent for instance. All STREE structures need to be initialized in such a case too. The future of this is to have the client GUI first create a new type master and then create an instance. Any locked node or line draws onto canvas in the selected fg and bg color, which is that of the locking stream. Different windows may have different such colors. New OS dependent functions to create and close a thread and a semaphore, as well as functions to lock and unlock a semaphore. Locking a semaphore is called with a timeout value so that the thread resumes when the timeout value expires if the lock cannot be acquired. Locking a semaphore under windows is more difficult as the current thread must still handle windows messages even when it's blocked waiting on a semaphore. A fairly complex MessageLoop function waits for messages, timeouts or semaphores and handles events without returning.

Improved ApiError to get Windows errors.


96 Tuesday, August 19, 2003

Fixes to the Real-time Interlock.

GUI {

TUserThread and TLoadThread are type masters that the GUI registers with the Core for tracking reasons.

MemoPrintMesage is now called asynchronously and needed to look-up the current thread to find the user interface pointer. A fix to TIsNodeClient::ChildInserted to only create a proxy node if the current one belongs to a Scope tree. Various real-time protections along the proxy trees. PleaseUnlock is obsolete. Fixes to the Progress function. DirectX canvas is now protected by PeekDispatch spinlocks. TUserThread waits in a spinlock for the construction or the destruction of the associated GUI. The Refresh Schematic timer is now protected against re-entry by a skipper. Loader thread is a registered type master and constructs a stream that locks onto the loader thread. Tray cannot make sense of the vector of forms, so it calls the new CloseAll function to do it. This recurses for the Forms vector taking note of the pointers, and destroys the User Interface forms upon backtracking.

}

Reintroducing the bitmap shape. A bitmap node loads its bitmap on set_Name calls. Name may temporarily be null.

The TSuperThread class of POSIX is obsolete. I think that the whole category will eventually become obsolete.

The Execute method of thread was promoted to stream. STREAM is THREAD for all intents and purposes. We're still keeping THREAD as a type master registration point. Initialization done on Open also needs to be done in INIT for all the STREAM descendents that don't do a stream Open, like the MEMORY_STREAM. The new STREAM::Open will spawn a thread if the mode flag is -2. Streams needed a SetTypeMaster (default constructor) to lock and unlock the stream onto the current thread. This was needed to lock streams constructed without initialization, such as from the serializer.

NODE uses GetShape to give descendents a chance to override. It is now used by constants and selectors to return a different shape while changing the meaning of the Shape field. Selectors may now be deemed to be bitmaps. There was a problem updating proxies on an un-intercepted set_Information call. Changes to NODE::SerializeNode to restore coordinate space if lock cannot be acquired. I can't remember why did we Unlock with lock promotion for a sub-node that gets deselected. The Change notifications should happen after the change was done. Unlike get_From/get_To, GetEndpoint is called under the assumption that the Enterprise is correct. The new GetEndpoint spins for the Pair[index].Parent to be non null. Spinning is done around a PeekDispatch call that the OS wrapper implements. Upgraded all drawing code to use GetEndpoint instead of get_From/get_To wherever the presumption of structure integrity held.

Resolved the multitude of real-time problems with updating them by adding the Options field to OBJECT with its ofProxyUpdate flag. The new PROXY_UPDATE macro uses this bit as a spin-lock to serialize proxy updates. There are far less crashes as a consequence. A new STREE::HasCycles function that is exponential in effort, and we're not keen on using much.

A problem was found with un-initialized pivots where the VMTs weren't set.

Object semaphores are constructed and destructed in SetTypeMaster now, since not everyone calls INIT (serializor doesn't). Not all INITs are called over zeroed memory (user created unregistered instances don't). Name and Information needed to be cleared. OBJECT::Free first locks the solid object interface. Simplified and corrected OBJECT::Lock. Semaphores are created with the object, and not on demand as before. Unlock informs the parent interlock of changes. Interlock change notifications are no longer promoted by TREE along the scope hierarchy because the interlock tree and the scope tree may have deadlocks between eachother.

There's a new strategy in the Core whereby the clients may register their own type masters. This is now used by the new GUI to create type masters for the TUserThread and TLoadThread. Consequently the TypeMastersVector needed to provide extra space beyond the CLSID_MAX_ TYPEID. There's also an attempt in this version to draw selector faces regardless of the dialer of the relations. A problem was found and corrected in the dialer where selectors were created on WRITE. It became evident with timer triggered redraws that multiplied the selectors. Type Masters may be bitmaps now. This state is incompatible with instance creation: a typemaster with bitmap cannot have instances. Other types may still aggregate it, though.

Resolved the matter of stack Progress indication. New OS dependent functions: PeekDispatch, ClaimProxyUpdateSpinLock, ReleaseProxyUpdateSpinLock, OS_ResumeThread, OS_SuspendThread. Corrections to the MessageLoop that was wrongly interpreting the results from the OS. OS_CreateSemaphore doesn't claim the semaphore now.


97 Monday, November 03, 2003

GUI {

It seems that the Tray Form must be visible upon exiting the Navigator to avoid a crash in VCL.

The Canvas should be read-only to avoid deleting it and crashing. We're now setting the font for the Canvas to that of the PaintBox of the Schematic Editor.

The Navigator doesn't allow operations on selectors. A new SelectedNode function returns the deepest node along a glove's finger that isn't a selector. Selectors may still be picked, but only for wiring operations.

}

Must draw the frame of a selected node even if the frame drawing flag is off. A fix to drawing a wire to a node when the bendpoint is too close to the contour. A fix to drawing wires to a node when its cable ends in the same node. This happens with inter-level wiring when A is child of B and A wires to B.

The dialer is shown only for a selected relation. Dialer is unnecessary for simple relations.


98 Wednesday, February 04, 2004

CORE {

A fix to the scaling of subwires within selector endpoints.

Changed the dialer logic to allow relations to be selected as fingertips of a glove of selectors. Added the "selection" parameter to the serializor. The grammar and this new selection parameters should merge as soon as the theory of text compiling changes to be relation-driven.

}


99 Friday, February 13, 2004

The current node needed to be fully unlocked before storing it.

CORE {

A good deal of work to make bendpoints coordinates relative to the cable's endpoint node.

Another matrix is passed to the line serializor and to the MoveBy routine to reflect the previous transform matrix for bendpoints.

Added another flag to STREAM::String to say that we want text with colored background. This makes line labels more readable.

Fixes to the node serializor:

  1. Recursively unlocking a node that is being unselected.
  2. Demoting the umbrella as the children serializor was descending was a mistake.
  3. Showing endpoint arrows only when the detail level permits.

Added another flag to Unlock to say that we only want to unlock the sublocks and not the current object.

Changed the logic selecting bendpoints and segments. The lock information field of a wire holds the recursive count of bendpoints for all the cables. Changes to line serializor and to MoveBy to accommodate.

Changes to the cable drawing algorithm:

  1. Not drawing the cable's control line unless selected
  2. Picking the cable anywhere inside the polygon
  3. Bending the cable and reducing the bends by picking on subwires.
  4. Selecting wires comming out of cables as interlocks of the cable's endpoint.
  5. Corrected the inter wire gap calculations

The logic that draws wires locked by another thread in a different colour may have suffered.

}


100 Saturday, February 14, 2004

CORE {

Discovered a flaw in the Depth Limited Serializor that was wrongly clipping an extra depth level, and another flaw that runs deeper in the system during binary loads that wasn't linking subnodes to previously loaded parent nodes. Changed the order of chaining children and parents in STREE::Serialize, but the real cure is to insert the stream into the morphing structure first and then expand it, rather than load and then insert with autobind.

Work on recursive cables. OPERATOR::Serialize is new and it first draws the wires of the operator with the graphical overrides of the instance wire, then draws the wire, by calling LINE::Serialize.

}

GUI {

Improved node delete logic that discriminates between deletions inside the ScopeView or deletions in the SchematicEditor.

}


101 Sunday, February 15, 2004

CORE {

Another line flag was introduced to say that the cable should free its wires starting with the last bendpoint. This logic creates more readable diagrams whenever the convergence of subwires to the end-node is too tight. It is also a way to pretend that the subwires aren't held inside a cable.

Line drawing algorithm now uses the TransformEndpointCoordinates function before recursing the subwires that come out of a cable. This function should also be used by line MoveBy.

A correction to RELATION::set_Cardinality to have it destroy any selectors whose index lies beyond the new cardinality value.

}

GUI {

A new ScopeExpand method that first collapses the scope tree, then searches for a scope path to start from Root and end in the node passed as argument to the function. Logic to auto-center the current or the selected node, to get the schematic editor to always show the designs centered. There are still a few problems with this algorithm. A global flag to tell the schematic editor's scroll bars not to redraw the design while auto-centering. Added another panel in the Parameter sheet to show the fully qualified name of the type of the selected item. Clicking on the panel will reload that type into the schematic editor. Added a response handle for the RELATION Editor Master Button to load the relation's supplier type into the schematic editor. A fix to LoadEditors to sense the operator type.

}


102 Tuesday, February 17, 2004

GUI {

Another control with 4 arrows for the TREE editor that does (should do) promotion, demotion, and re-order of nodes in the tree.

A relation editor master button event handler that changes the parenthood of the relation's field between supplier and client.

A line editor master button event handler that changes the direction of the connection. Only valid for relations. It triggers morphism.

New type flags: flUnion and flMultilateralRelation to be used by Core.

}

CORE {

New type flags: flUnion and flMultilateralRelation. flUnion instructs the TYPE::GetSize logic to return the max of all relations of that type.

Fixed a problem with not being able to dial into a field when the type and the relation are on different scope levels, by recursing the serializor for traversal with umbrella=to.

The level parameter is passed through the line serializor (reintroduced the former detail) as it was needed to recurse cables.

}


103 Wednesday, February 18, 2004

GUI {

An attempt to turn the whole navigator into a DLL that may be invoked from another application through a simple CreateTray call. The DLL only has two exports. CreateTray and SpinTray. Unfortunately the Navigator DLL crashes upon constructing the TUserInterface form.

}

CORE {

TYPE implements another flForward flag, and has released the boolean (char) Forward field for future uses. The new flag also obsoletes the relation's ISFORWARD flag which is now available for reuse.

Fixed a problem with the TopLimit and recursing relation fields. Notable is that field "corridors" to other types are not subject to top limiting.

Added a test to k5core that opens a DLL and calls the CreateTray export function. I couldn't make it without including Windows.h.

}


104 Saturday, February 21, 2004

Added retained mode support to the Enterprise Navigator.

The DirectX surface driver was separated from the GUI UserInterface into its own DLL at UserInt.CPP revision 93. The DxD module was modeled and generated. A new TestDraw is implemented for DirectX that uses retained mode to draw some cubes.

Did a preliminary code cycle over the GUI to extract the VCL model and the imported non-class types. The important issue here is that nonclasses work only if they are flat listed in the <unspecfed> category. Having things such as TForm in a category of <unspecfed> would fail to generate code. Added some DirectX declarations to the <unspecfd>. The new DxD module keeps its interface clean of any such declarations that would oblige the client to include DirectX header files.


105 Sunday, February 22, 2004

GUI {

Must split the GUI into quadrants.

The Scope Viewer was segregated from the TUserInterface class. The new TForm_ScopeView has the toolbar for load, store, new, print, cut, copy, paste, sort, delete to control the tree view control. All the expand, select, etc. logic was moved here from UserInt.

Extended the viewer with a list view that replaces the Type Master Reporter. The columns are resized from a header control. The TIsNodeClient also made it here from UserInt and extended to load values in the Reporter. Double click over the Reporter loads the CurrentNode's type back into the scope viewer. The reporter shows the fully qualified path as the type name.

}


106 Sunday, February 22, 2004

GUI {

Must split the GUI into quadrants. Worked on separating the Schematic Editor out of TUserInterface. The new form is TForm_SchematicEditor. Most logic was transferred.

}


107 Monday, February 23, 2004

GUI {

Demoted the Enterprise Navigator into a subdirectory of Navigator.

}


108 Thursday, February 26, 2004

GUI{

The new GUI strategy is to have a generic GUI frame with docking capabilities and external forms that may dock into the panels provided by the GUI frame. There are 7 panels provided that may be shown or hidden and may also be resized:

  1. Top and bottom panels for user supplied toolbar and status bar.
  2. Upper-left, lower-left, upper-right, and lower right panels that may dock external forms.
  3. Debug log panel.

The Enterprise Navigator was split into modules that may build as DLLs and are functional in simple EXEs.

The Navigator's program group now builds the old interface as well as two versions of the new one which has been split into modules. The new modules are: About, K5Components, GuiFrame, Runtime, SchematicEditor, ScopeView, SyntaxOrientedEditor, Tray, TypeViewer. Refined the build procedure to use precompiled headers. All GUI components need to include the Navigator.h header file. Moved our modified color grid to k5components. Resolved the hugely difficult issue of docking TTreeView controls: the drag and drop logic was deleting and recreating the tree nodes thus corrupting the cycle links between node and data. The solution was to override CreateWnd and DestroyWnd (needed a new component for that) and get these two to call user installable events.

Added a pascal function to SEDIT and importing it from the tray.

}

CORE{

Added the new OBJECT::pDebugMessage function pointer for user installable message loggers. The function pointer defaults to a printf to console. The static function OBJECT::DebugMessage takes format specifiers and channels them to the installable pointer. A new class DBGMSGFMT is superseding DBGMSG, and takes format arguments, allowing for formatted ENTER clauses. The DBG macro is now using the new installable debug subsystem. Had trouble with the tested/untested macros in the interface and debuglog.

Added the fail variable to the TDxdControlBase::Construct function. We needed to inspect it and not take any pointers to an auto-destroying DXD object.

}


109 Friday, February 27, 2004

GUI{

Further simplified the generic tray and the application main body. The tray is common to all programs. Each main program may descend from the tray and instantiate that specialized version. An app creates a few more actions, then calls MakeMenu to show them with the tray icon. There has been a lot of trouble with BCD's control of the application's main form and was resolved with CreateTray and SpinTray calls. These two may live in an external module which is our new main() equivalent where we can do the real application specific work using the multi relation.

All our forms are now TForm_Relation descendents specialized to auto-link to a relation. The relation is the actual client program operating over and through the relation's end-points. The base relation defines the interface for inserting/removing forms and bringing to the top of the relation of a certain endpoint. The descendent GUI multi-relations (user applications) will need to implement the interface to provide actual storage for the multi-relation. The forms at the endpoints communicate by de-referencing the multi-relation and by calling the defined methods of the generic or special relation.

Reactivated some Scope View, Schematic Editor, Type Viewer interaction through the TMultiRelation class. We're just doing simple selection now. Enhanced the TForm_Gui to use a multilateral relation to manage the active endpoints if more than one of the same type of form is docked. The frame senses tab changes on the generic docking stations and moves the corresponding relation's endpoint to the top. We can now have more than one tree viewer, and more than one editor for the selection of a tree node.

}


110 Thursday, March 04, 2004

CORE{

The new DEBUG_BASE class was promoted to the top of the model and is shared by all other classes. It only contains statics and commons, so it doesn't add to the data size of any descendent. DEBUG_BASE implements the formatted output to a log device through an installable pointer. The base class also implements the OpenEditor channel through an installable pointer. The DEBUG_BASE::pOpenEditor function pointer is the beginning of the Viewer/Editor registration hierarchy. A type viewer library would intercept this pointer and filter-out class IDs to match those it knows to open an editor for. Added a new abstract class TNotifyRelation which is the ancestor of TNullRelation and doesn't introduce any notion of GUI or forms. This base class adds to a core object the ability to cooperate with a GUI through a multiple relation that it only knows in terms of its non-directional notification capabilities.

Changed the inclusion system to exclude the specification of the root directory.

Corrections to the sample TTrayIcon component that crashed when popping-up the menu in our new builds that have no default application form. Changed the tray icon to show the application's hint rather than its own.

OBJECT now descends from DEBUG_BASE and gets debug capabilities from there. TInterface descends from DEBUG_BASE. The DXD driver although it is not an object has now gained debug trace capabilities and logs the DirectX errors to the log.

}

GUI{

All forms are now declared as PACKAGE and they do not include the form pointer.

The Tray moved from its own subdirectory into the GuiFrame library. Since it deals with two most important tray menu options: create new frame and exit, it now implements two virtual methods to create and to destroy the main GUI. Also, there was a great deal of trouble showing the icon of the tray in a DLL build. We resorted to LoadIcon directly from the EXE resources to circumvent the problem.

}


111 Wednesday, March 10, 2004

GUI: {

A fix to the trace log facility: we need to activate it so that it auto-scrolls. Also, page changes will now activate their content.

Had to change the CONTROLS.HPP header file of TWinControl and declare OnKeyPress, OnKeyUp and OnKeyDown as publics, so that we may assign them to response handlers of the Schematic Editor's form. It's the only way to turn a non-focusable control into one that accepts keyboard input. In the keyboard handler of the Shematic Editor we handle some key combinations that allow us to select nodes with the KBD, and using the new ChangeSelection function that first unlocks the outgoing node, then locks the incoming one. Reduced the unused arguments of SelectNode.

Resolved the tremendously difficult matter of activation and focus of controls when the form is docked in another form:

  1. Forms weren't considered as active controls of their parent form. Had to change that, and devised the TNullRelation::Activate method to reset the active path of controls along the tree.
  2. The ScrollBox_SchematicEditor needs to be marked as the active control of the form at construct time, so that the dock manager knows what to do with it.
  3. The ScrollBox of the schematic isn't a focus control, so normally it doesn't grab the focus from any other focused control on the form or onto any other form from the hierarchy. To change that and give the TForm_SchematicEditor focus we now use the multilateral relation's new Activate method, which sets the sub-form as the active control of the form owning the relation. Then we say that on the local form, the ScrollBox of the SE is the active control.
  4. We save and restore the active control through the alterations of visibility of the SE toolbars.

Selecting nodes in the SE reloads the Type Viewer through the Relation::Reload method. The Reload method may be called by any endpoint and will act on the remaining endpoints accordingly. An absolute reload is done by calling Reload with a null endpoint.

Re-instated the copy of the schematic to clipboard as a picture.

Selecting nodes on the Schematic Editor now auto-expands the Scope Viewer. Scope View nodes that were auto-expanded will be auto-collapsed. Manually expanded scope nodes will stay expanded unless they get selected in the schematic editor.

Added the options of the DxD driver that allow the 3D demo to spin.

Added the Expert Reductor flags to the toolbar of the SchematicEditor. Added the popup menu that integrates all Schematic Editor's actions. Resolved the mouse wheel problems in the schematic editor.

The new TForm_ScpeView::ScopeFindExpand method is one better than ScopeExpand as it doesn't collapse the tree if it can find a path or a partial path to the specified node. The result is a much more stable scope tree with respect to the schematic editor.

Added a popup menu to the Scope View with all the actions. Added the TopLimit and Depth buttons to the toolbar of the Scope View. They drive the binary serializor and the cut and paste operations. Drawing the fully qualified type names for the scope nodes in reverse order as required by the left-hand justification.

Had to explicitly call the initialization/shutdown of the scope view in the monolithic version.

}

CORE: {

Added TREE::RelativePathReverse.

DXD has now flags and options. When ofAnimate3D is set the TestDraw function is involved and draws a demo 3D scene. We construct the retained mode driver only when the ofEnable3D is set. These hardware ZBuffered surfaces are much more expensive than the double buffer we usually require to alleviate the flicker of the schematic editor. To export the static data DxD needed to be built with __export and be used with __imort.

}


112 Saturday, March 20, 2004

CORE{

TCanvasStream is slightly faster in that it compares the PEN and BRUSH values and doesn't change them if it doesn't need to.

There's a new TCanvas method that renders a type to a bitmap, thus producing icons. This was a long awaited for result.

STREAM has another flag that will prevent drawing text on canvas. This is to draw bitmaps for types.

Hierarchies are now building their own bitmaps by drawing their public face in reference mode, down to a detail level of 1, on a little canvas. The Schematic Editor has a control to preview it.

TYPE has another flag: flInvalidBitmap. This is set by the upward Change method. The new TYPE::Change method sets the flInvalidBitmap flag. TYPE has a new ImageCache field where a bitmap is painted whenever GetFace is called and the type face is invalid.

OBJECT has a new way of updating the proxies using a skipper flag in the UPDATE_PROXIES macro. The flag shouldn't be global, but on a per-instance basis. OBJECT hasn't got a Flag field yet, and it uses the MSB of InterlockData to bar re-entry through UPDATE_PROXIES. Destructor of OBJECTS spinlocks before de-initializing the Proxy structure.

STREE was split into STREE and STREE_BASE which was promoted from ABSTRACT to BASE for uses outside the scope of the Morphism Core. OBJECT descends now from OBJECT_BASE which is new and promoted to BASE. OBJECT_BASE interacts with STREE_BASE. The Name, Information, and Proxies were promoted from OBJECT to OBJECT_BASE for general use.

TREE::DONE now does a Remove. I can't see how I've overlooked that.

We had to know the first entries of the VMT by ordinal to implement the static-virtual calls of the TypeMasters. A chain of enums deal with it know and descend from one another.

A new mechanism: TREE::LockCone that recursively locks a tree (the direct tree aspect of the Enterprise) and TREE::UnlockCone that does the opposite. The new functions are used in the NODE pick correlation algorithm. The old scheme of unlocking the entire cone of interlock was destroying the relationship between Canvas and CurrentThread in a multi-thread and mutual lock.

Refined the ApiError reporting in the OS_ global functions in TEST.CPP. Core reports errors to the debug log.

}


113 Tuesday, March 23, 2004

Major changes:

  1. A re-evaluation of the interaction between Core and GUI involving the extraction of typeface bitmaps has lead to new mechanisms that create and update bitmaps on the fly.
  2. New logic sensing when core objects get dirty.
  3. Core now permits installable type master modules. This is one of the most important milestones of this research and a four years overdue promise. Type master libraries are DLLs of a special build, with one export that returns the topmost type master category for the caller to use. So far only the FaxCore cluster was turned into such a module. Type master libraries are released when the core deletes their top category. A great deal of care went into un-mapping the DLL without loosing the return path of the caller's stack.
  4. GUI is now spawned on its own thread. The dynamics of the system is therefore a lot more complex.
  5. GUI allows dropping foreign objects over the Scope Viewer. This is the new file loader. A discrimination process recognizes type libraries and mounts them into the back bone of the Morphic Core.

MORPHIC CORE {

A new FreeTypeLibrary function is now exported by the core to carefully circumvent the problem of un-mapping a module and not having any code to return to. This function simulates in C what would otherwise be an assembler function that pops the stack by two nesting levels, prepares the stack frame for a call to FreeLibrary, changes the return address to that recorded three levels before, and returns in a special way. The function must be called with care.

The Morphism Core exports a "C" function that destroys the specified node. This is to support the command line tester, which will grow to be the OS's command shell as soon as we will have format information built into the Core. The tester has a new command to load a type library, and another command to delete the current node. Deleting a type library super-node is to unmount it.

Visibility was a char field of TREE using only 2 bits. We needed flags for OBJECT and therefore promoted the field to OBJECT as generic Flags with 6 bits defined in the bit-field. Visibility is now a 2 bit-field. Had to do that to maintain the current size of objects and not have to invalidate the file format. We needed a flag for the Proxy mechanism to mark the object instance for deletion and during the destruction of proxies. The former scheme of using the MSB of the InterlockData field was sometimes either corrupting or being corrupted by the interlock. Added the GetTypeID virtual method to TREE. It returns -1, but type libraries override it.

}

GUI {

TNullRelation is now a circular list that tracks inserts/deletes, and the topmost (active) relation. Had to do that for multi frames. BuildFrontier is more complex now, Changed, Refresh and GetDragSource are new abstract methods of TNullRelation.

The Gui Frame now collects all the main menus of the components that dock into it, reducing duplicates. The menus are re-evaluated whenever the null relation's Changed method is called.

Removing a relation endpoint recalculates the main menu.

Activate calls onto the co-relation will update the Active field of the circular list of relations. Focusing an endpoint will register a locker object with the Scope Viewer.

The MultiRelation's Reload method opens the parameter editor/viewer associated with the (re)loaded object, if one is registered, and docks the new editor into the lower-left panel of the GUI. Refresh is a new co-relation used between the Scope Viewer and the Schematic Editor. GetDragSource is a new co-relation between Scope and Schematic.

The Enterprise Navigator runs on its own thread now. Any further component is constructed on the GUI thread of the last GUI frame using a command interface between the Tray and the Gui thread.

Type Viewer has a working TypeName button now.

Corrected the Gauge component to not draw degenerated pies.

TTreeViewWnd was enhanced with the custom draw methods that call the OnInfoDraw event. The tree view is now accepting foreign objects dropped from the outer world via the Shell API in response to WM_DROPFILES message. The OnDropFile gets called once per each file dropped.

ScopeView shows a Gauge control with the stack consumption. ScopeView uses a more abstract Locker object rather than the CanvasStream. The Schematic Editor gives the Scope Viewer its Canvas to use as Locker, upon co-relation-reload. Scope view initialises its Load Thread Type Master once per process. The LoadThread is more complex now and can be configured in the constructor with file type and file name. A new file type is ftTypeLibrary to support dropping in type libraries for registration. The new Scope View doesn't cache any bitmaps for type masters, rather it constructs them on the fly, during CustomOnDrawItem. Corrected the drad and drop operations in the Scope Viewer and towards the Schematic. If there's a current thread in the sense of the THREAD class, then the scope view takes a proxy to it. The proxies have a new way of drawing their progress by means of a mosquito marker that spins around the branch box.

The Schematic Editor has more static flags (the former global options) to configure the Expert Scope Reductor. It has a comprehensive set of actions and a large local menu for all its operations. The main menu gets merged with any other when the schematic Editor is docked to the Abstract Gui Frame. There's a new mechanism of previewing the type face bitmap of the current schematic inside a 16x16 toolbar button. A new action was itroduced to take advantage of Core's new LockCone function. This action selects a trunk cone between the serializor's limits. The redraw of the schematic is now compounded with the workings of a refresh timer. The keyboard now goes to a secret memo control that gets focused instead of the Schematic's scroll box, which isn't a focus control. The keyboard commands cycle through the nodal arborescence. Upon construction and destruction, the Schematic Editor shares or retracts its canvas from any other co-related component that may use it for interlock. Added an action to log the degree of cross-reference for the current node.

}

Type Libraries {

This is the new way. Components bound into DLLs are mountable into the Core's backbone at runtime. The FaxCore is now built as a type master library. A type Library may also be made of modules: Core+GUI. A GUI library links to its associated Core library and overrides the OpenEditor chain of pointers to add itself to the filter set of the Core. A party may decide to invoke the viewer/editor of a given object by calling the OBJECT_BASE::OpenEditor static method, which strts the filtering process to locate the appropriate viewer. The OpenEditor was extended with two new arguments: Owner and Relation. FaxCore now initialises by also loading a GUI library from a module with the same name and the TED extension if one is present. The fax core is constructed in the Core+GUI split model.

The FaxCore Type Library uses an ID range over 110. New classes: TFaxType is a pivot that creates fax object instances. CAT_FaxCore is the exported type master category. NewObject is static. Every FaxType has a static Type field to its type master.

The new FaxCore uses the BASE_STRE class supplied by BASE and exported by the Enterprise. get_Next, get_Prev, get_Parent and get_Children of TFaxObject are now wrappers of BASE_STREE, and not wrappers of VCL TTreeNode. Protection to several instances where null pointers now develop.

ShutdownFaxCore needs to erase all traces of the fax type masters from the OS backbone.

}

FaxClient {

New logic to delete the registry keys of the printer we want to uninstall. Logic that attempts to stop and restart the spool service and dependents.

New actions and a main menu for the Fax Directory View. The menu and actions are merged by the new Gui Frame.

Adapting to the new Gui Frame that owns a circular co-relation.

}


114 Friday, March 26, 2004

GUI {

Config.H defines TNullRelation and TForm_Relation. The implementation now goes to Config.CPP which needs to be shared by all GUI DLLs and EXEs.

Attempted logic to re-task a GUI frame from one thread to another. This is very difficult, not ready, and currently disabled.

The rebar controls are not docking sites any more.

Had to slow down the daemons gy a factor of 16. They introduced an unacceptable performance degradation.

}

CORE {

A correction in TESTS.CPP to the MessageLoop and OS_LockSemaphore tandem. The return codes were wrong.

Moved FreeTypeLibrary and ApiError to BASE, so that it may be seen by the whole world. Moved the code to load a library into the BASE as the new OBJECT_BASE::MountLibrary method. The new OBJECT_BASE::Kernel is a static pointer to a registered kernel that may be loaded or unloaded.

Fixed the problem of deleting an object that locks other objects. The Interlock tree must not self destroy, but self unchain. Only the direct tree aspect of the Core may self destroy actually. Added a flag to STREE_BASE::DONE to tell it not to delete the container object.

Commenced work to decouple the Morphism Core from the Navigator GUI. So far, the Type Viewer, Runtime, and Schematic Editor were reworked to not link with OS.DLL and use virtual methods into the Core. More methods of the Core needed to be virtual. BASE is the onion center and has a pointer called uKernel. The OS registers here as the last step of the constructor. Any user of the kernel may de-reference it by linking to BASE and using this pointer. Begun the work of turning the INIT constructors into virtual methods. The Core is getting closer to becoming a Type Master Library. Getting the static Options of all classes is becoming a design problem as it needs to be done through virtual get/set functions. Changed the logic of Type pivots to be derived from the Kernel through macros.

Pseudo-assembler in debugsrv.cpp to avoid the .000 problem.

Added the bitmaps to the FaxTypeLib.DLL as resources. Type faces are now served from the Type Library itself, rather than the GUI. Moved the CAT_FaxCore out of the fax modules and into the library Entry Point module, as it won't be easily ported to Linux. A map of CLSIDs for all Fax types. COMMON_DECLARATIONS and Type pivot for the Fax Server class.

}


115 Tuesday, April 06, 2004

Highlights:

  1. Core is a Type Master Library now.
  2. Abstract proxies to OBJECT_BASE. Everything done through the micro-kernel.
  3. GUI doesn't link to Core anymore.
  4. GUI threads are forked. See BASE for words on the new fork mechanism. No need to initialize the scope view anymore.
  5. Another Tray action that attaches or detaches the GUI to and from Core.

GUI{

  1. Another constructor of relation forms that take a window handle as argument. This is to allow one module to be built (for debug) with VCL as library.
  2. Core is a Type Master Library now.
  3. Abstract proxies to OBJECT_BASE. Everything done through the micro-kernel.
  4. GUI doesn't link to Core anymore.
  5. The GUI Form has no knowledge of threads these days, and has a Command field where it may accept instructions from the tray.
  6. The Enterprise Navigator forks its main GUI frames. See BASE for words on the new fork mechanism. No need to initialize the scope view anymore. The GUI form is created on a forked thread. The form doesn't maintain a Thread field anymore, instead it has a Command field that is programmable from any other thread. The tray uses this command field for the topmost GUI form (which it now finds in the relation tree owned by the tray) to synchronize the injection of new viewers into the frame and the closing of those frames. The actual thread code for each Frame is owned by the Enterprise Multi Relation.
  7. Another Tray action that attaches or detaches the GUI to and from Core.
  8. GUI Frames are now linked into a tree and have the tray as common parent.
  9. Closing the tray while frames are still open has changed and is considerably more rugged now.
  10. Work to protect the Schematic Editor against a sudden unload of the Kernel.
  11. The Scope View achieves its asynchronous loader by forking inside the new OpenFile function. TLoadThread is obsolete, and replaced by the much more graceful fork.
  12. The Scope View is using the Core through a virtual interface and no linking. Core is a Type Master Library now. Protections were required against a sudden loss of Kernel.
  13. There is a new and generalized proxy mechanism. The Scope View uses proxies that only ever need to see the OBJECT_BASE virtual table. The scope Expanding and Collapsed have moved inside Proxy, as a step towards moving the whole lot into the TTreeViewWnd reusable component.

}

CORE{

  1. Core is a Type Master Library that may be loaded and unloaded into the BASE micro kernel.
  2. Base connects to Kernel over several services: CreateThread, CreateForkThread, GetCurrentThread, etc.
  3. BASE implements a new and much more powerful multi-tasking mechanism: the fork. The current fork is somewhat better than the UNIX counter-part in that it doesn't require the creator to wait. In turn the weakness is that only the first-level stack frame is available as copy to the child thread, so references and "this" need to be copied. The fork is a complex algorithm that works even in the absence of the Core and creates normal system threads, or, if the Core is registered, it would create THREAD constructs that may then be used for interlock. Fork is an assembler simulation that finds and copies the creator's stack frame onto the new stack of the child. The frame copy is serialized by means of a spinlock so that the creator doesn't exit and destroy the original stack frame before the child had a chance to get its context data. Our forks take non-UNIX arguments such as Name and stack size.
  4. All common and static functions called from outside the Core had to be turned to virtual methods to allow the loose runtime coupling and dynamic linking.
  5. Certain (few) constructors that were called by GUI had to be wrapped into corresponding CAT_MORPH virtual functions.
  6. Core doesn't include Windows.h anymore.
  7. Added the memory management category to the directory.
  8. The Free upwards call end-up at the CAT_MORPH level where it is resolved to FreeTypeLibrary.
  9. Mutex was a serious problem. We're only allocating them at lock time and we're destroying them upon unlock. For that we had to construct them on the locker's thread, under a unique name concatenating the internal name plus the ascii handle. The real-rime of the system is much better now. All the thread-related business was promoted to BASE.
  10. Destroying the categories means to first disown the sub-scopes and type masters. DO_DONE sets TypeInfo.Object to 0.
  11. INIT_TYPE_MASTER macro was promoted for general use.
  12. UPDATE_PROXIES was promoted to BASE.
  13. OBJECT_BASE has its own reasons for calling the Change notification channel. The EChange enum now inherits from BASE.
  14. There is a new hierarchy of tree aspects starting from BASE. OBJECT_BASE starts the policies necessary for registering and querying the various STREE cross-referenciators.
  15. Corrections to the destructor of OBJECT to keep the instance locked for longer, until all proxies were destroyed.
  16. Object Lock and Unlock had to go deeper to BASE level in order to notify these events to proxies. OBJECT_BASE::VLock and VUnlock are new for this purpose.
  17. Interlocking is now done without Locks, who has now vanished. Its sole purpose was to allow the proxies to update on this important event. The proxies now update based on activity on the various tree aspects of their servers, including but not limited to the interlock.
  18. Important fixes to the Unlock logic in the real-time environment: must not either conically or recursively unlock threads locked onto the object.
  19. Core doesn't call any UPDATE_PROXY anymore.
  20. The default destructor looks to de-register the Kernel from the BASE.
  21. The default destructor destroys all the proxies to that object. Logic is protected by spin-locks.
  22. GetTreeAspect is a virtual that any type overloads to add its particular trees.
  23. THREAD overrides the Stream's SetTypeMaster to avoid interlocking onto the current thread, since it is the current thread.
  24. There are new STREAM and THREAD constructors, as well as an API to create threads with a user supplied Execute function. This is to support the new FORK mechanism.
  25. Changed the LINE AutoBind algorithm to only set the parent of a line once. This greatly improved the loading of models where lines (relations) were first bound to endpoints, then bound into cables. The GUI Scope View receives less notifications this way.
  26. BASE has changed considerably. It implements the new UNIX-like fork mechanism. It abstracts the proxy update mechanism so that no other type in the system should need to intercept any virtual method other than VLock, VUnlock, the tree notifications Inserting, Removed, Reordered, and the most general Change.
  27. STREE operations are exclusive through the use of thread sensitive LOCK/UNLOCK. This is to help the real-time safety of the system.
  28. A fix to the STREE destructor which was inadvertently clearing the container pointer.
  29. A fix to the DxD driver to not call PeekDispatch while locked, because this allows for the handling of WM_CLOSE message and recursion.
  30. Correction to the creation/destruction of the top category of the FaxCore.
  31. A protection against a possible null cardinality in model files.

}


116 Saturday, April 17, 2004

CORE {

  1. OBJECT::INIT has changed to adhere to the policy of the new Morphic Memory Manager. MMM manages free blocks, used blocks, and object instances. Of those, only free and used blocks are linked in their TypeInfo aspects. These kind of blocks are of the new AMORPHOUS class type. We say that a Type Master instantiates objects, whereas an ALLOCATOR instantiates amorphous blocks. Once instantiated, Type Master instances are detached form the allocator that supplied the amorphous memory block, and then linked to their respective type masters. The allocator perceives these objects as uncontrolled gaps that are hopped over during MemAlloc and MemFree searches. It is the responsibility of each type master to destroy instances by first turning them back into AMORPHOUS blocks, and return them to the Allocator, and finally free them. The process of returning an amorphous block to the allocator needs to respect the strict order of block addresses. The allocator's Instances pointer is the last block of highest address, generally the available free block. The instances next pointer follows the used or free blocks decreasingly. OBJECT instances are initialized uncontained (or free-floating) and without handle data.
  2. The new Morphic Memory Manager (level 2) has the option of either allocating a constant adjacent to the NODE header - case in which the data handle may not (yet) be resized, or to allocate a header that handles a cluster allocated elsewhere. Handle reallocation only works in the latter scheme.
  3. The new Morphic Memory Manager doesn't allocate used space headers for type instances. Therefore, there is a difference between deleting objects and deleting any other raw data. The delete operator can't be instructed to make this difference, consequently all free-floating type instances must be deleted using the Free method which in turn will cal the MMM's Reclaim method. All symbols that fail the grammar reduction during serialization must be deleted using the Free method.
  4. Destruction of contained instances has changed again: the container pointer is required to be valid during the DONE calls. DO_DONE has reverted back to clearing the container pointer after destruction has finished.
  5. The ~OBJECT destructor calls DONE, but still needs to unchain the TypeInfo. This is because the new DONE destructor only unlinks the TypeInfo of contained objects, and leaves the free-floating ones untouched.
  6. I seemed to have had to sacrifice the rather useful TypeInfo change notifications, don't know why.
  7. Since the new STREE_BASE::INIT preserves the Children, FlatCount, and TotalCount fields to support the construction of morphic pivots, we need to clear those in all the SetTypeMaster methods of those types that introduce new tree aspects.
  8. Needed to circle object instances backward too. The new Morphic Memory Manager does that. Implemented a get_PrevInstance method. IsObject is a common test used by the Morphic Memory Manager, or anyone else who may have doubts whether a block is typed or is amorphous.
  9. The new Core MMM demands that every type master has a virtual static GetInstanceSize method that generally returns sizeof(type). The MMM uses this method to calculate data offsets and to allocate object headers into the managed data space.
  10. Unfortunately we now need to delete the name/info before calling SetTypeMaster(0) in OBJECT::DONE. This means that lengthy destruction logic cannot be identified in the debugger.
  11. Added another test to OBJECT that looks for instance chains broken by a null Prev or null Next. It has been useful to catch some flaws in the Morphic Memory Manager. Added more tests to STREE::SelfTest that hunt for selfish children in both directions.
  12. ALLOCATOR is a new addition to the Core. It implements the second level of the Morphic Memory Manager. It does MemAlloc, MemFree, MemRealloc and instance Reclaim. It also has a prototype Grow method. ALLOCATOR also has a comprehensive SelfTest that dumps the allocation structures and shows all the relationships to the type masters that have allocated instances from this managed space.
  13. CAT_ABSTRACT contains the Allocator type master which is a morphic pivot. Meanwhile, CAT_ABSTRACT now descends from ALLOCATOR and has therefore became the system's primary memory manager. CAT_MORPH now descends from CAT_ABSTRACT rather than to contain it, so the two types are one, and CAT_MORPH is the system's main memory manager.
  14. The AMORPHOUS object descends from OBJECT only so that it stays within the design space of the Morphism Core. It should really be exported at the top of the hierarchy so that it has the smallest size. It currently overrides DONE and set_Name to allow amorphous blocks to be named with static strings.
  15. The Core may build without the MORPHIC_MEMORY defined and use external ANSI malloc/free/realloc. In such case, the construction/destruction scheme is as before. All object instances are newed from the RTL heap and have another header that's specific to the memory manager.
  16. TYPE::CreateInstance changed to make use of MMM and not allocate any header. When called with a size specified, the MMM allocates a block that is header+size and the new instance will have a handle pointing after the header. Constants or selectors may also be created by creating instances of a 0 size, then setting the MaxSize and Handle to an external cluster.
  17. TYPE::DeleteInstance is different for MMM because MMM doesn't follow the containment chain to free the ultimate container when a portion of it is freed. This means that not all type instances may be deleted and the instance iteration logic had to account for that.
  18. One major conceptual problem related to type master pivots. The TYPE::DescendsFrom method is ineffective for a morphic pivot. Once again: a Morphic Pivot is an object of its own type. A Morphic Pivot is abstract. Morphic Pivots may be complex types that have become pivots after a series of specializations. In essence they have the characteristics of some other type they last descended from before declaring (themselves) as pivots. What can we say about the ancestry of such a type master that has severed the inheritance chain and now points to itself? One certain thing is that they are still Types, and must have once had TYPE up the ancestry. To faithfully answer to DescendsFrom questions, each pivot should override the method.
  19. CAT_MORPH registers itself as the BASE Kernel upon initialization and de-registers on destruction. We kept having problems with the RPC Interface so it had to be disabled. The Autoload stream had to be contained.
  20. TSuperKernel has changed for the new Morphic Memory Manager. The Kernel is located at the very end of the un-initialized data segment between the HeapStart and HeapCurrent variables, followed by a configurable 32M of empty BSS to extend the MMM into. We now say that the Core has a morphic BSS.
  21. The Core doesn't need to initialize the ANSI library because the Thread Local Storage is provided from the BASE micro kernel.
  22. The Core may build in the following ways:
    1. Morphic Memory Management, my own ANSI library. This is the "independent" way. OS.DLL starts with our own VMT startup module and only ever links with our pieces, plus the foreign IMPORT32 and ODBC32. In this model, however, our ANSI maths support is very slow. We're emulating the FPU for portability and we're about 60 times slower doing graphics in the Schematic Editor than using Borland maths. Also there's something wrong calculating line^circle and our wires stop at a higher radius than the circle. Will fix that.
    2. Non-morphic memory management, my own ANSI library.
    3. Morphic Memory Management, Borland runtime libraries for everything else than malloc/free/realloc. The fastest.
    4. Non-morphic memory management with Borland runtime libraries used for everything.
  23. Added conversion logic to SerializePObj to salvage the binary files that had references to CAT_ABSTRACT.
  24. Added another argument to the STREAM constructor to ask the Core to cache it. It can't work for statically allocated stream instances.
  25. Caught a destruction problem with NODEs that may be constants or selectors. Handle reallocation for NODE is only possible if the header and instance data are not subsequent.
  26. Removed the ExpressionStatement production from the Compiler because it can't pass the integrity test.
  27. The Rose loader has changed quite significantly. TRoseBaseList is finally a TREE descendent. That simplified things, and has the added benefit of passing the morphic integrity test. TRoseBaseObject had changed the way it does CreateInstance as a consequence of the introduction of the Morphic Memory Manager into the Core. TRoseBaseObject is a morphic type and it wants to allocate extra instance space for data fields that aren't yet defined. Consequently, TRoseBaseObject overrides GetInstanceSize and adds some 300 bytes to it. Hunted-down all the many little memory leaks of the Rose Morphic Compiler resulting from token manipulations.

}

ANSI {

The Thread Local Storage of the ANSI library was demoted from the ANSI library to BASE. BASE is an OS dependent layer.

The console window history buffer and height are now preserved.

The errno variable now goes into the initialized data segment. It links last and mustn't end the BSS which is now controlled by ANNA K5's Morphic Memory Manager.

This ANSI library leaves malloc, free and realloc undefined, and to be supplied by the user.

}

BASE {

EndForkThread had to be public.

STREE_BASE Children, FlatCount and TotalCount are now preserved by the STREE_INIT. This is because the STREE_BASE may be partly initialized before construction for certain uses such as TypeInfo.

Corrections to the InsertBefore algorithm that left the parent's counts unchanged. The algorithm wasn't expecting a child of one tree to be inserted before a child of another.

The Thread Local Storage of the ANSI library was demoted from the ANSI library to BASE, which is an OS dependent layer.

The Tiny Memory Manager was demoted from the Morphic Memory Manager to BASE as it is OS dependent. It is needed when a client application hasn't yet initialized a memory manager, yet it must dynamically allocate something. The Tiny Memory Manager isn't intended to reclaim space. We only use it to serve memory during the startup of Borland/Microsoft runtime libraries that unfortunately do mallocs for file locking structures.

}


117 Monday, April 19, 2004

CORE {

  1. The Morphic Memory Manager was refined to create minimal block headers. Initially, the amorphous blocks were OBJECT descendents and were too big (117 bytes). By re-aligning the hierarchy to have OBJECT_BASE descend from AMORPHOUS, we got the header size down to sizeof(STREE_BASE)+VMT=40. The ALLOCATOR, OBJECT, and TYPE were affected. AMORPHOUS was promoted from Hierarchy to the base micro kernel and exported.
  2. GetTypeInfo is a new method of amorphous blocks. TYPE overrides this to supply statistics.
  3. IsObjectInstance is superseded. Since AMORPHOUS is at the top of the hierarchy and deals with registration of instances to their type-masters, the get_NextInstance, get_PrevInstance, get_Instances, get/set_Handle have been promoted there, and turned to common. BASE always links into the user application. The OBJECT::TypeInfo field moved to the AMORPHOUS class to begin the hierarchy. The ISObject method returns the type master pointer.
  4. Caught a problem in the memory manager. It was always chopping the last and largest free block during MemAlloc. No reuse was done as a result. I'm searching the free list back to front now, looking for an exact fit. If there isn't enough space for the requested size plus an amorphous block header, then the search continues. We cannot give the caller more memory than he requested, for we don't want to maintain another field with the real size.
  5. CheckAPIError was the same as ApiError exported by BASE. PeekDispatch should not be duplicated.

}

GUI {

Restored the ScopeViewer's detail grid based on K5's new GetTypeInfo method.

}


118 Wednesday, April 21, 2004

CORE {

The Core is the second layer of the Onion. The installable Type Master libraries are layer 3. They need to be independent of the operating system and be portable. The issue of porting certain components which are tightly bound to a given host OS to other platforms will be dealt with at a later stage in a more generic way. Right now, all the foreign calls that the Core or its 3-rd layer type masters make to the host OS are funneled through the static HOST_OS family of classes. Our ANSI library is OS independent except for a number of low-level calls to do with threads, files, comms, and memory, which are bridged to the host OS through in the new HostOS.h/CPP modules, part of the BASE micro kernel. We're now saying that the BASE micro-kernel is the minimal piece that everyone needs to link to. Removed all the free OS_ functions that BASE was exporting. BASE is exporting the host bound functionality behind the following classes: HOST_OS_COMM dealing with sockets, HOST_OS_RT dealing with threads, processes, semaphores and modules, HOST_OS_GDI that deals with fonts, brushes, pens, lines, etc, and HOST_OS_SQL that wraps the SQL functionality. BASE.CPP is now windows independent, and any foreign calls that it does go through the HOST_OS class. BASE.DLL builds either with Borland's libraries and startup, or with my own ANSI and the minimum ever STARTLIB.ASM startup module which just calls DllEntryPoint.

Re-enabled the RPC interface. All host dependent calls now funnel through HOST_OS exported by BASE.

Ran into a startup problem with the tiny memory allocator. where malloc starts to be called by Borland's startup module even before the BASE.DLL is initialized. Had to move the initialization of the Tiny Heap upon the first ever call to malloc. Another problem right behind the first is that upon a free, the block must first be filtered to see which memory manager maps it, and delete the block with respect to that particular manager. base_free now returns true if the given block falls within its range.

Added a simple Fork test to the k5core command shell.

}

ANSI {

Promoted certain abstract and platform independent functions from bridge to conio..

Protected strcpy against a null source pointer. Bridge.CPP has become windows independent after all the OS dependent functions have moved to BASE.

}


119 Friday, April 23, 2004

Major results:

  1. The entire system is Windows free. All system calls either go through ANSI or through the new HOST_OS classes.
  2. The entire system was ported to MSVC6 toolset where it builds as a monolith and runs just fine. The monolith entails my ANSI, the BASE micro kernel, the Morphic Core, and the FaxCore type masters.
  3. FaxCore is Windows free.
  4. My own starup modules.
  5. Ported the Morphism Core to MSVC tool-chain.

CORE {

  1. Ported to MSVC tool-chain.
  2. The DllEntryPoint is defined at build time so that multiple such modules may be gathered into one monolithic EXE.
  3. The HOST_OS modules implements all the ANSI functions that need binding to the platform. Some of them are just defined and not implemented.
  4. Enhanced the HOST_OS funnel with more socket functions.
  5. LINE::set_To isn't testing to prevent line-to-line wiring.
  6. Had to rework for MSVC6 the static virtual methods of OBJECT such as PointerArithmetic. The difficulty was to find a way of taking pointers to methods, casting them, and calling them for a different object instance.
  7. One very strange thing is that the MSVC6 compiler emits data into the segments (BSS) back to front. BSS looks more like a stack than like a heap. That kind of throws-out my plan to do morphic BSS for MSVC6. Changing the HeapEnd address manipulation logic.
  8. For MSVC6 __thread is __declspec(thread) The or/and words seem to be reserved under MSVC6.
  9. NODE::getChildren renamed to get_LastChild. Correcting the argument types of the constructor in the CONNECTOR template.
  10. Reworked the Fork mechanism for MSVC6 which generates code with a different prologue/epilogue code. Using a __naked function prototype for MSVC6.
  11. Enhanced the K5CORE command line tester to build monolithically with all components and call their DllInits.
  12. The RPC debug server is stubbed-out, as the core doesn't want to include Windows anymore.
  13. Ported the VMT startup module to MASM as a dialect that TASM also understands. Catering for the differences in the name mangling schemes of Borland and MS.
  14. The Interface has some assembler that was disabled for a build other than Borland.
  15. Changed the __export/__import for __declspec.
  16. Ported the Morphism Core to MSVC tool-chain, which is stricter in enum to int conversions.
  17. Stack calculations aren't ready for the MSVC6 build.

}

FAX{

  1. FaxCore is OS independent now. All socket communications are funneled through the HOST_OS_COMM class.
  2. The monolithic client builds using the BASE as a DLL.
  3. MSVC6 doesn't allow virtual methods to differ in their return type. All tree operations such as get_Parent had to be renamed to get_FaxParent. I never understood this kind of C++.
  4. Functions such as filecopy and CheckFileSize were promoted from the fax core to BASE, as they are ANSI-like by nature.
  5. All connection code is now wrapped, simplified, and abstracted by funneling through the HOST_OS_COMMS interface.
  6. MSVC6 limits string constants to some length that the prologue of the fax cover was exceeding. Splitting.
  7. Had to do some discrimination for different ANSI libraries for walking the file system looking for cover sheets.

}

ANSI {

  1. Ported to MSVC tool-chain.
  2. Added the S_IRUSR, etc definitions to STAT.H.
  3. Added PROCESS.H collecting POSIX definitions and prototypes.
  4. Moved the NULL definition out of TIME.H and into STDDEF.H. Added prototypes for C++ Borland and MSVC6 builds.
  5. Moved strdup from STDLIB.H to STRING.H
  6. Moved the definition of NULL to STDDEF.H.
  7. Added the _exit function.
  8. Added STDIN_FILENO, STDOUT_FILENO, and STDERR_FILENO for UNIX compliance.
  9. Moved close to IO.H, and mkdir to DIRENT.H. Defined _mkdir to mkdir macro.
  10. Wired in the _LONGBITS, _SHRTBITS, and _INTBITS to their Intel values. Defined wchar_t and redefined size_t.
  11. Added va_start, va_arg, and va_end macros for MSVC6.
  12. Slightly different MSVC6 scoping rules requires all functions to be introduced in advance of their use.
  13. MEMORY.H is an inclusion gate into STRING.H and MEM.H for cross-platform compatibility.
  14. Allowing math function prototypes for C++ with __BORLAND__ and MS_CVER.
  15. Prototyped rename, eof, dup, dup2, close, mkfifo, chmod, and access into IO.H as required by ANSI.
  16. Isolating the #if (sizeof(...)!=xxx) pre-processor asserts only for the Borland compiler. MSVC doesn't seem to allow it.
  17. Moved getch and putch to the HOST_OS bridge and bound them to the OS. Changed the meaning of get_or_test_ch to _kbhit and reworked the conin to be blocking and work correctly in the absence of the Borland startup. getche now echoes, and gets uses getch with putch instead of getche.
  18. Ported the assembler constructs of _ftol to MASM.
  19. Renounced the FILE related rename for the handle-related ANSI variant.
  20. Added the O_NONBLOCK 0 definition to stub UNIX.
  21. Declared errno as static to avoid linker clashes between multiple files that include errno.h.
  22. Collected some used directory related constructs into the DIRENT.H header file as required by cross-platform buids.
  23. Added dir.h to satisfy the Borland builds.
  24. Redistributed the function prototypes across the ANSI headers according to the standard. Ongoing work.
  25. Added fltused, and purecall to Bridge. Moved exit to HOSTOS and binding it to do a process termination. Moved unlink to the HOSTOS bridge as stub. _setargv__ was removed as nobody seems to want it anymore.

}


120 Friday, April 30, 2004

Ported the Core to GNU tools.

  1. Ported some IO to the GNU inline assembler, but couldn't work out the expressions.
  2. Cannot have multiple deletes under GNU.
  3. Source files had to be renamed to begin with capital letter then lowercase. The path separator used in inclusion system had to change from \ to the UNIX style /.
  4. Had to change the startups to allow for GNU entry points.
  5. Including memory.h instead of mem.h
  6. No declspec for GNU.
  7. Disabled the strupr in path calculations in the main parser.

121 Tuesday, May 04, 2004

Ported the Core to Linux.

The format and layout of the Virtual Method Table varies between Borland, MSVC, Linux and Cygwin as follows:

  1. Under Borland we know it's an array of longs, passing this on the stack.
  2. Under MSVC, VMT is an array of longs, but because this is passed as register, we need a member pointer de-reference to manually call the functions through the VMT.
  3. Under Linux, the VMT is a vector of longs, cannot make any assumption as to the model of passing "this", so we need to use member pointer de-references. Further-more, the first two VMT entries are reserved, and our methods always start from the third entry.
  4. Under Cygwin, the VMT is a vector of __int64 with the first one reserved. Must translate the method pointer into a normal function taking this on the stack. The low order long is null.

122 Thursday, May 06, 2004

Under Linux we had to give-up the spinlocks for atomizing the STREE methods. This is because of the ill effect of calling GetCurrentThread. DoReturn needs to do a pthread_exit under Linux. Sleeping once inside the Fork spinlock loop.

The Fork logic was ported to GNU inline assembler, for those who can understand my thoroughly constrained pain.

Caught and fixed a bug in the HOST_OS::Accept method that was corrupting the socket. Validated the Fork logic for Linux. Implemented the mutex logic for Linux using dynamically allocated pthread_mutex_t structures. These are structures of about 20 bytes under Linux as opposed to the handle used by Cygwin. HOST_OS::Sleep does an ANSI sleep under Linux, which is not right, but would have to do for now.

Implemented the thread create/destroy and the mutex create/destroy/lock/unlock logic for Linux.


123 Saturday, May 08, 2004

Under UNIX new processes are created with fork(), and this is the only way. Fork clones the address space of the caller process on a per-hit basis. This is a deep reaching difference to Windows. We had to level that difference if we wanted to be symmetric. Her's what we do: firstly we don't fork processes for Linux but we branch the calling thread which is almost similar in the cloning aspects of the address space, but different when it comes to exec a foreign program. Doing an exec from within any thread of the current thread will overwrite and substitute the entire calling process even though the calling process had other threads that should perhaps be kept alive. I'm not going to argue here against this UNIX design which is perhaps sound if this is what you're used to. Our logical problem, however, is that we need the calling process to survive, yet branch another thread that runs an external program, and this is a legitimate request. We have to avoid the exec function that overwrites the calling process with the child. Might want to use the spawn family of calls that do that, but, big surprise, none such functions thing in Linux. We implement one for Linux that is essentially a call to fork with a waitpd loop at the end. So we've swapped the non portable fork+exec to the portable branch+spawn. Spawn is called with underscores for MSVC runtime library, so we had to have a discrimination there as well.

FaxCore{

  1. The build directory layout is now different.
  2. The cover sheet sources now live in the FaxCore subdirectory where the server is built. The exe is built one directory above the FaxCore sources. The logic of finding the cover sheets had to change to strip the exe name from argv[0] and append FaxCore, then the cover sheet name.
  3. Swapped the non portable fork+exec to the portable branch+spawn, to level-out one major difference between Unix and Windows.
  4. Turned the MAKECOVERSHEET protocol command to be passive. The server may be called to do this from any client without it being logged into the server. We're thus taking steps towards having an all-passive server.

}


124 Wednesday, May 12, 2004

CORE{

BC502 very badly corrupts the INTRFACE module depending on how the project tree is modified. We need to build this module first and clean, otherwise the image will have incorrect relocations.

Several fixes to the core were required after the port to Linux to restore former Navigator functionality:

  1. The base FreeTypeLibrary function was going through the HOST_OS::ModuleFree method which, unlike FreeLibrary, isn't __stdcall. The whole logic behind arranging the stack so that we get back to the caller of OBJECT_BASE::Free after the code module was unloaded was thus compromised. Had to move FreeTypeLibrary to HostOs and treat it as an OS dependent function that calls FreeLibrary directly.
  2. Deleting the connector of a relation was not recursing upwards towards the container relation. Generally speaking the Morphic Memory Manager had to allow the OBJECT::Free method to test the instance containment and recurse upward towards the largest container object that isn't an Allocator.
  3. A crash condition was found and fixed when GetTypeInfo was called for an unnamed type.
  4. Traced a memory leak to calling close on a thread handle. The file oriented close would then malloc some locking structure, thus polluting the morphic heap.
  5. Traced and fixed another memory leak in the destruction of memory streams. The buffer is deleted and then the handle must be set to -1 rather than 0 so that the inherited stream destructor doesn't attempt a close call, which would malloc some locking structure that pollutes the morphic heap.
  6. Found and fixed a problem in the morphic allocator where a free block was wrongly appended to the end of the allocator's list whenever chopped to use a piece of it. The allocator's instance chain must be kept strongly ordered.
  7. Enhanced the allocator's self test with calculating the sub-object count.
  8. During the port to MSVC who disallows overloading methods that differ in their return type, we introduced NODE::get_LastChild to replace the former NODE::get_Children. The nodes were, however, wrongly using them them interchangeably resulting in crashes on the Schematic Editor during tiling operations for instance. Corrected. Line selection was wrong due to another porting problem that left an un-initialized segment variable.

}

GUI{

Enhanced the Scope View OpenFile method so that the possible new loader thread is compared against the caller's thread before a proxy is constructed for it.

}


125 Wednesday, May 12, 2004

CORE{

Corrected problems:

  1. The Morphic Memory Manager is enabled for GNU builds as well.
  2. Names for windows mutexes can't be longer than a certain. Reducing the names so that the serializor doesn't fault with mutex creation errors.
  3. Changed OBJECT::IsLocked to do easier rejects before harder ones. Added a protection against long recursive interlocks.
  4. The Rose model loader was wrong because the open files could not be found due to case and slash differences.
  5. Enhanced the TREE:Find algorithm to compare names case insensitive, and with insensitive path separator characters. This, of course, was needed for nesting file streams.
  6. In certain conditions SerializeExport was locking the stream onto itself resulting in a recursive interlock structure. Added a protection against that.
  7. Argv[0] will contain the installation root directory instead of the program name, for all the secondary main tests.
  8. I had to go back to a based INTRFACE.DLL module, which is a very annoying shame.

}


126 Sunday, May 16, 2004

BASE {

  1. Designed and implemented the INI file manager with the following components: INI_TREE, INI_SECTION, INI_FIELD, and INI_FILE.
  2. Fixed the build problems relating to spawn under MSVC.

}

CORE{

  1. Added the INI load/store test to the main console.
  2. Added a command to kill the Server to the main console.

}

GUI {

}

FaxCore {

  1. Caught a hang condition when the fax server cannot find the coversheet directory. The client is not informed and remains hanging on reading the socket. Enhanced the protocol with an appropriate code and getting the client to abort the suction of the covershhet names.
  2. Using our new INI_FILE manager.
  3. The FaxTypeLib.DLL + FaxTypeLib.TED arrangement induced many build problems as they both generated FaxTypeLib.lib. The fax core is now FaxCoreLib.DLL. Changed and corrected all user dlls.
  4. FaxClient is using the new INI_FILE. Corrected the drag and drop logic. Corrected the form destruction to save the FAD to the INI file. Disabled the Listener and the Uncoiler because they gave linker problems. Simplified GenerateTestData.
  5. FaxTrayMain doesn't link to FaxCoreLib or to FaxTypeLib.
  6. Added a protection to the MultiRelation selector in case the FaxTypeLibrary cannot be loaded and no editors are available.
  7. Obsoleted functions that implemented a tree using vectors. Added TFaxObject::get_FaxFirst. Turned the fax server into a FaxClient descendent in view of actually merging the two.
  8. The TThread stub class was promoted to Fax.h
  9. Server now exports its socket, so that an external agent may close it to get the server out of the listening loop.
  10. The main program for the server reads and writes the INI file. Also, a global pointer to the server is taken. When called again from the same process, the command line may terminate the server by forcing the server's socket to close.

}


127 Wednesday, May 26, 2004

BASE{

  1. A new base mechanism to persist objects to INI files. An enterprise stores to an INI file using a serailizor. The INI is regarded as a sequence of sections. The first level of serialization is supplied by OBJECT_BASE::Persist workin in tandem with INI_FILE::PersistPObj. This scheme comes from K5's OBJECT::Serialize and STREAM::SerializePObj for binary files. The base mechanism is to recursively serialize all registered tree aspects of an object.
  2. Enhanced the INI_FILE functions with another parameter that specifies the separators in the file.
  3. The new INI persistence mechanism needs to know a vector of Types vector. The Core now exports it upon construction.
  4. The Core is now in charge with creating proxy nodes. That's done by calling a notification virtual method that must be intercepted by a proxy GUI node to create another one like itself.
  5. New OBJECT_BASE methods: Persist, GetTypeID, CreateInstance.
  6. The Core now specifies its various tree aspects for the generalized browser. The TypeInfo tree aspect only begins to make sense above the TYPE level.
  7. NODE overrides the base ExpandTree method to do a limiting expansion of the direct scope which is physically the same as the inherited tree aspect, but filtered only for nodes.
  8. LINE overrides the base ExpandTrees method to do special processing and expand the To and From tree aspects.
  9. CreateInstance returns a generic OBJECT_BASE pointer, one after the AMORPHOUS type.
  10. Corrected the set_Name proxy update channel.
  11. Capital module names.

}

GUI{

  1. The ScopeView is (almost) independent of the Morphic core, and does (most) things through the Proxy mechanism that was promoted to a Borland CBuilder component. Proxy.CPP and Proxy.H are now the Rose generated K5TreeView modules. The proxy flags were divided into flags and masks. The masks are abstract and supplied by the Morphic Core.
  2. ScopeView has another mechanism now for setting a node as the browsing root. Re-enabled the details grid.
  3. All GUI forms are now proxies to the micro kernel so that they may receive notifications such as to redraw.
  4. An interesting result of the new browser is the ability to view all other proxy nodes, and perhaps, delete them.
  5. TIsNodeClient was modeled and generated. Same for TK5TreeView.
  6. We're in the world of graphs, not trees. Graphs have cycles. They also have multiple arborescent aspects. The windows TreeNode is not enough to handle graphs. TIsProxyNode was the wrapper of TTreeNode that managed the objects of the Enterprise Kernel. The FaxCore is another such device that has just gained multiple tree aspects and thus needs the same kind of a proxy node or wrapper. Had to export the TIsNodeClient used by ANNA K5 and SATURN in a manner that is independent of the inner workings of K5 and Saturn. Had to devise tha abstract GetTreeAspects method that returns the count, the names and the offsets of all tree aspects of an OBJECT_BASE descendent. The Enterprise Kernel fills in these values for its (3 to 7) tree aspects so far. The FaxCore now uses 2 tree aspects. The GUI ScopeViewer or derivatives are now using the mask flags to specify what tree aspects they want to expand when the user clicks on a tree node. This component links to BASE.DLL and regards it as the first layer onion shell. For more details, please see the UKernel section of the K5 Concept Report.

}

FAXCORE{

  1. For the monolithic version the Fax Core didn't initialize due to the lack of DllEntry. Exported the init/shutdown methods of the core and acting on them in the WinMain of the monolith. This isn't a problem for the DLL build, but here we've got a more difficult problem of memory allocation/deallocation across different modules. This unbalanced scheme is used in the string send/receive. The monolith also needed to embed the resources of the FaxCoreLib.
  2. We're now using a TK5TreeView Borland CBuilder control instead of TTreeViewWnd to browse the FAD. The FAD is now doubly arborescent, and hence, cyclic. The new abstract browser was needed to gradually expand it.
  3. INI persistence is now called with a specific list of separators.
  4. The FAD viewer extracts a toolbar under the New menu with all the available types found in the FaxCore. The icons are served from the core's resources.
  5. FAD viewer has a simpler and more consistent drag&drop routine that creates references. Delete is also simpler.
  6. Total control over how FAD nodes are drawn.
  7. Needed enhancements to TFaxType to find a type pointer by ID or get the ID of a type. These are used in persistence and in extracting the bitmaps.
  8. TFaxObject has a new tree aspect to maintain its associations. An associate of a FAD node is a symbolic link of another object.
  9. The VCL TreeNode is obsolete now due to the abstract expansion mechanism.
  10. TFaxFile is a new but empty class.
  11. Corrections to the logic that iterates FAD branches for addresses.
  12. Simplified the TFaxQueue destructor that destroys reference nodes more simply now.
  13. A new and improuved INI persistence scheme.
  14. Simpler object creation logic.
  15. Core can now indefinitely expand the nodes.
  16. A FAD data generator that is more comprehensive and demonstrates the use of symbolic links to archive documents.
  17. Changed the icons of TFaxClient, TFaxDocument and TFaxQueue to something nicer.
  18. The Fax Core builds a vector of types and registers it with the uKernel.

}


128 Friday, May 28, 2004

BASE{

}

CORE{

  1. Added the TEditorStream type master to STREAMS. It wraps around a Notapad or RichText control. This module was relocated to STREAMS from the former UserInt GUI module.
  2. Changed the AutoBind/AutoRemove mechanism to work regardless of the type of parent cable. When cables are deleted during AutoRemove they naturally loose the bend-points and attributes. The ScopeView should refrain from inserting a newly loaded node with autobind enabled, because this would reset all cable attributes.
  3. Enhanced the OBJECT to trigger Change notifications on SetTypeMaster calls. These are an opportunity for the GUI to update the instance counts in an info grid.
  4. Had to disable the one-click selector pick correlation scheme for relations of a cardinality of one because we couldn't otherwise pick the relation itself in order to, say, delete it.
  5. Changed the significance of a NODE's tree aspects: the direct tree aspect is the low-level detail of a NODE that includes the subnodes as well as the line femurs.

}

GUI{

  1. The TK5TreeView was extended in terms of relating TIsNodeClient nodes to their corresponding info grid items for direct lookup. The scope view is now able to refresh the info grid upon each daemon step. There was a scenario where the TIsNodeClient::Change notification comes while a fork is in progress. The Fork may have been called from a GUI event handler which appears to be setting a critical region. Doing further GUI processing from another thread, such as de-referencing tree items, under these circumstances results in a deadlock.. We're now querying the state of the ForkSpinLock and only update the node proxies if the micro-kernel isn't forking. Another problem was regarding the masks of the proxies of children with respect to the mask of the parent proxy. The issue here is that the parent proxy and its children may have tree aspects that differ semantically, so their expansion masks are not comparable. We're not doing the AND anymore.
  2. Changed the ScopeView to new nodes as children of the selected node. The ScopeView should consistently use a multi-selection algorithm that is independent of the Schematic Editor's canvas interlock.
  3. The Icon Preview logic was triggered on every node move, and yielded nasty recursions through the serializor. This resulted in several over-draws. Introduced logic to enable/disable the Icon Preview by clicking on the icon view.
  4. Enhanced the Multilateral Relation Editor with GUI controls for wrapping the text, compiling the text, general flags, and a status bar that shows the attributes of the paragraph where the cursor is. The text limit of the RTF viewer was maximised through a private message, so the text is no longer truncated. The action of compile is only crunching the lines of the RTF and logs it, together with paragraph and attributes, to the trace log.

}


129 Thursday, June 10, 2004

CORE{

  1. Introduced the concept of aliases for object names. Storage is still the same, but alias names are separated by the pipe | character. OBJECT_BASE::GetName would only return the first name without aliases.
  2. Added some new logic to load UML associations from a Rose model file. These associations resemble the K5 multilateral relations. We needed to port those back to containment relationships to salvage the legal system's design, so we aren't currently loading associations as multilateral relations, but rather like normal binary ones.
  3. Added time_t to the base model in expectation of the new time-stamping algorithms.
  4. Added a few virtual methods to TREE to be used for navigation. They are volatile and would be lost upon the next code generation.

}

GUI{

  1. Enhanced the Multilateral Relation Editor with the compiler of documentation. This is an important step.
  2. Enhanced the GUI of the Multilateral Relation Editor with logic to disable the OK button if no changes were made, added the text wrap flag, logic to pick the first alias for the tab name since object names have now got aliases.
  3. The names for the connections of the multi-relations are calculated to either be the user-configured name for the line, or if this is null, to be the endpoint name without its aliases. This scheme creates much clearer lists of connections.
  4. Enhanced the I/O browsers of the Multilateral Relation Editor so that they highlight the central text with any alias found.
  5. Better delete logic for the current multi relation loaded in the Multilateral Relation Editor.
  6. Logic to show the total number of lines and the current cursor position in the Multilateral Relation Editor.
  7. Added the Font selection button to the Multilateral Relation Editor. Added logic to resize the I/O panels and hide them if the Multilateral Relation Editor docks in a small place.
  8. The multi-relation editor responds to the hard reload events of the Enterprise.
  9. Added the Keep action/button to the Multilateral Relation Editor. Its role is to determine the editor to not respond to refocus events sent through the Enterprise Multi-relation.
  10. Added the search button and the logic behind for the Multilateral Relation Editor. Currently the search procedure isn't refocusing the RichText viewer around the new and possible hidden position of the caret.
  11. Added the InfoView tool, which is a simple viewer/editor for the currently selected object. It responds to soft reloads.
  12. We wanted a way to recognize the tabs other than by name since tabs of different types may be loaded with the same node. Enhanced the multilateral Enterprise relation with methods to get/set the bitmaps of these tabs. Each GUI tool such as ScopeView, SchematicEditor, InfoView, MultirelationView, etc. have a bitmap and an index they register with the generic GUI form by. Enhanced the tray to also display these bitmaps associated with the actions and the popup menu to create new instances of the tools. Enhanced the generic GUI with logic that puts bitmaps on tabs, if the upper layers define them
  13. Merging the main menu of the GUI frame was failing to get all the menus when one type of tool wasn't instantiated. Corrected by adding a GetEndpointsCount to the null relation. Descendents supply the actual number.
  14. Enhanced the ScopeView to issue a "soft reload" to the EnterpriseMultirelation whenever the selection changes. Certain editors such as the Schematic Editor are programmed to listen to hard reloads, others such as the InfoView listen to soft reloads. The difference is in how many times the user clicks on a node to trigger a hard reload.
  15. ScopeView isn't double buffered.
  16. Had to disable the info-grid for general use because it's too dangerous during loads and stores. The K5TreeView inserts/removes new lines into the grid depending on how new proxy nodes are created or deleted. The logic depends on the expansion of the scope tree and makes use of VCL's AbsoluteIndex of TTreeNode. Calculating the absolute index is logic that involves the entire tree and may stumble upon branches that are destroying. We cannot afford exceptions to be thrown from browsing the VCL tree because the serializor cannot be reset and the interlock structures will remain dirty and compromise your work. We need to go back to an info grid that doesn't need absolute tree node indexes.
  17. The info grid is enabled from a checkbox in the ScopeView's toolbar. When active, the grid is updating much softer now, without deleting the items, which, again, would be dangerous given the extreme noise that the Morphic Core makes during its operation
  18. The tools are forms with separate actions and menus, and with potentially overlapping shortcut sets. Often I wanted to delete the current node in the Relation Editor, and lost the entire tree (without saving my work) because the ScopeView responded instead. We had to implement a menu re-ordering scheme so that the commands and shortcuts of the last form that the user clicked on have precedence.
  19. Added some protections against null current node in the Schematic Editor. Some wrong logic that reset the CanvasStream and thus made the schematic editor unusable was corrected.
  20. Created a 16x16 daemon icon alongside with the former 32x32 version. The tray form has an image list now for the popup menu.
  21. Added the About tile of the Enterprise Navigator, a feature that was dropped since UserInt.
  22. Allowing the tools to be launched in the absence of a generic GUI container.

}

COMPILER {

  1. The new scope compiler/generator works on a RichEdit stream as the input band.
  2. The language that it recognizes is defined by the following grammar:
    1. <GOAL>::=<Node>
    2. <Node>::=Information <ChildrenList>
    3. <ChildrenList>::=HeadingLabel Name <Node> <ChildrenList> | Lambda

where Name and Information are the attribute of any object, and HeadingLabel is either a numbered sequence or a string from set of labels configured in a list of precedence.

The compiler is tailored to understand any Word document that has a hierarchy of headings, which is either outline-numbered or labeled. The result of the compilation is a morphic scope hierarchy. The text generator does the opposite and generates the documentation/code for a scope hierarchy. The process of compile generate is cyclic. The user may configure the depth of generation/compilation as the deepest child that would be reached.

  1. This compiler/generator is in its infancy stages and will eventually merge with the Syntax Oriented Editor.

}

FAXCORE{

  1. We needed the INI2OBJ macros promoted to Fax.h.
  2. Added the concept of time to FaxCore. All fax objects now have a CreationTime and a LastModified field. They go to persistence. An object is considered to have been created when first constructed, not when loaded from persistence. The Touch method updates the LastModified time stamp. The touch granularity is the second.
  3. Added new types: TFaxFifo, TFaxFile, TFaxProxy. The new Fifo is an independent object segregated from TFaxClient to deal with uncoiling and generation of new documents in the queue where it is installed.
  4. EvaluatorDelay, PrinterName and PrinterDriverName were demoted from TFaxClient and are a Server feature now.
  5. The Client software constructs a Fax Server object now according to the principle of symmetry.
  6. There's a dynamically created new toolbar behind the New button of the GUI. All types supply their bitmaps there.
  7. FaxClient doesn't need to link to FaxTypeLib.
  8. Added editors for the new TFaxFile, TFaxFifo, TFaxProxy, and TFaxServer types.
  9. Picking a symbolic link in the FAD view will bring-up the editor for the target FAD node.
  10. All FAD editors now descend from a common TForm_TFaxEdit class that introduces the Object field.
  11. Enhanced the otherwise useless TFaxQueue editor to list the queue content in terms of names and time stamps.

}


130 Sunday, June 13, 2004

GUI{

Implemented the Type Merging Principle of the Morphism Core.

  1. Enhanced the Multilateral Relation Editor with type merging logic. The editor has now got 3 modes of operation: normal, merge, and differential, selectable from a new 3-way radio-button. The differential isn't yet implemented. In normal mode the editor's content is reloaded when the user drops another node over the name input line. In merge mode, the node dropped will be merged into the one already loaded. This process involves several steps as follows:
    1. The name of the source type (the one being dropped) becomes an alias of the name of the type already loaded.
    2. The information fields of both source and destination types will be concatenated. This operation is made difficult by the complex nature of the RTF document format that Information may be described in. To merge Information we're using the clipboard and the center relation editor.
    3. Must re-insert all the sub-types of the source type into the destination type. This process will trigger the auto-bind logic and change the cabling of the over-all structure, thus affecting a cone of types into the global scope starting from the common parent of the source and destination types. Only types in a non-degenerate scope relationship may be merged (this is a morphic principle).
    4. We need to merge the inputs/outputs of the source type with the inputs of the destination type. This process is delicate due to the general problem of iteration-removal. To make matters worse, the inputs (and outputs all the same) may be structured cables containing wires also pointing to and/or from the source type. The algorithm that relinquishes relation endpoints is recursive in the second degree.
    5. Finally delete the orphaned source type.

Another important matter is implementing the differential. Automatic type merging is one of the goals that the Morphic Core constantly seeks to fulfill. Merge automation also implies automatic (unaided) discrimination of types.

This logic really belongs to the Morphic Core, but the compilers for Morphic Assembler and Documents aren't yet merged.

  1. Resolved the matter of automatic scrolling towards the cursor in the Search process of the Relation Editor, and by fortunate consequence in the Log View as well.
  2. One huge and nasty problem just popped-up: the CBuilder front-ends must also be compiled with the option of int enums. Functions such as GetLine were failing because of this.
  3. The Relation Editor reloads faster now, by ignoring tree-node change events that in normal operation would reload the left/right relation information views.
  4. Made icons for Exit, Browser, and Help to be used for the Tray popup menu. Added logic to set one 16x16 icon for the tray and another 32x32 for the rest of Windows.

}


131 Monday, June 21, 2004

BASE {

Object names may have aliases. The GUI should not use compound names. Added a global alias index that may be set by calling SetCurrentAlias. GetAlias is new and may isolate a specified alias from the compound name irrespective of the CurrentAlias value. GetName now calls GetAlias with the CurrentAlias value. I'm also using GetAlias in the TREE::Find routine.

}

CORE {

  1. LINE_Find had to be a virtual function of CAT_ABSTRACT so that the GUI may use it.
  2. Enhanced the TREE::Find method to isolate the first alias from a compound name.

}

GUI{

  1. Windows is periodically sending an update notification to all applications so that they may reload their INI file. In our case VCL was reloading the system font used for help hints. The TApplication was attempting to recreate the hint window and generally crashed because ANNA K5 has a multi-threaded GUI and the hint window would have not been created by the main process thread. Only the thread that created a window can destroy it. First I had to disable those notifications. That wasn't enough to prevent the hints from crashing. Eventually resolved the matter at the TForm_Relation level by disabling hints for the form that looses focus and enabling them for the form gaining focus.
  2. The abstract GUI now has a page control as the top panel so that you may drop forms in there too. Fixed the small size of the tab sheet label when just one page remains in the page control. Fixed the drag and drop logic to set the dropped form as the active page.
  3. The InfoView was enhanced to activate itself to the top of the multi relation when clicked.
  4. The About form displays the version number of the Morphism Core.
  5. Added one more test to the menu merging routine of the TNullRelation to avoid merging the menus of more than one instance of the same form.
  6. Added the following bitmaps for toolbars:
    1. Keep and XRef to the Relation Editor
    2. Plus and Search to the Scope View.
    3. Keep and Apply to the Schematic Editor
  7. Had to disable the Tray code that collected all the Help requests to the main help file. We want the individual forms to be able to override the main help file and supply their own.
  8. Added RtfOle.C to the Relation Editor in an attempt to embed objects into the Information field.
  9. Enhanced the Schematic Editor with the Keep button to make it deaf to the refocus/reload notifications. Corrections to the AutoCenter logic.
  10. Enhancements to the Multi-relation Editor.
    1. The GUI has changed to have resizable status panels for the center editor.
    2. New actions: InfoRelations, Load an external file.
    3. A hidden temporary RichEdit control to be used in merging information fields.
    4. New configuration flags to: skip over lines, and to change the meaning of the I/O panels from endpoint info to link justification. By default, the relation editor doesn't stop on intermediary lines and jumps from concept to concept. The jump-over-lines flag may be ticked-off to load a line into the center editor.
    5. New radio-buttons to express the new XRef operation modes of drag & drop onto the Relation Editor.
    6. Changing the Name or Info must check the Keep button so that the relation editor stops responding to refocus events until the content is deliberately saved or discarded.
    7. New threads: Compiler and XRef maker.
    8. ReDoc.
    9. Reduced/unified some code loading the I/O panels.
    10. All editors are loaded in the non-modified state, and the Okay button is disabled until any one of them is dirty.
  11. The behavior of the Relation editor.
    1. The new compiler can automatically create aliases for the objects it loads. The grammar has two new specifiers: \a to construct an alias from the next line of text, and \i to ignore the first word of the alias line.
    2. The progress of the Compiler/Generator are monitored in a Gauge control, and in absolute terms into one of the status panels. The compiler gauge is only visible during compilation. It builds its 100% value from the total character size of the text.
    3. The compiler/generator is spawned from the tool button, and runs as a separate thread as long as the button is down. The compilation process may be canceled by depressing the button.
    4. The new compiler is now looking for the case-sensitive grammar leading tokens
    5. The descent and back-tracking logic of the compiler logic were corrected to allow incomplete paths. An incomplete path is something like a.b.e.g.h whereas the complete grammar path would be a.b.c.d.e.f.g.h. This is too complex to discuss here. Please refer to Concept Report for details on the compiler. The compiler may now correctly load legal text such as the Common Law and the Criminal Law.
    6. The IO cable viewers now discriminate between cables and relations and extract different bitmaps.
    7. Browsing through the cables of the I/O panels will highlight the info panels as follows: the base notion is found and lit into the center concept's info panel; the center concept is matched and lit into the base concept's info panel. The opposite cross is established between current concept and derived concept. Highlight should not attract a dirty state.
    8. The IO cable viewers are now checking each line in terms of their parenthood and endpoint, and only log the primary links. Any structured link may be browsed by descending into the primary cables. Destroying the IO cable viewer's content is much faster now as we hide and disable the panels first to avoid the refocusing noise.
    9. Correct values for cursor positions may only be read after the Rich Edit has interpreted the mouse down or the key-down events. We therefore update the status-bar on mouse-up and key-up.
    10. There are two new drag&drop modes: XRef and Reverse XRef. The cross-reference builder is a thread spawned from the XRef button. The button is down until the entire XRef of the current concept and dropped concept is built. The process may be cancelled by depressing the button. The XRef builder is monitored in the status-bar with a gauge control. The 100% of the gauge is calculated as the product of the type weight of both source and target concepts. The information XRef of the current concept may be destroyed by right clicking onto the XRef button.
    11. The XRef procedure will take a pair of nodes and math one's name and aliases into the other's information field. If a match is found then the two will be related through an information relation if they aren't related already. The owner of the relation is either the source or the target node depending on whether a direct or a reverse XRef was specified.
  12. The Relation Editor has its own help system that overrides the global one.
  13. An attempt is in progress for embedding OLE into the information fields.
  14. Enhancements to Scope View:
    1. Added a control to change the current alias for the names in the tree. This is a global variable that affects the entire Core system. The objects of the Morphism Core may have aliases. By default, the GUI displays object names in their main alias. Changing this value will refresh the scope tree with the new aliases for those nodes that have them or the last alias for those nodes that have less aliases than the value specified. This value affects the entire GUI, Schematic Editor, Multi-relation Editor, etc. The Sort button now works over the current alias.
    2. Added the Search functionality. There's a new button in the Scope's toolbar that launches the Search dialog. The string specified there is then used to find a descendent of the selected node. If one descendent node is found, then the scope will be auto-expanded to that node. The search cannot currently be repeated as that involves a large scale rework of the Core's find mechanisms.
    3. The Save process is now threaded just as the loader is.
    4. there's a new but unimplemented level-expand button.

}


132 Saturday, June 26, 2004

GUI {

  1. Enhanced the system with the ability to embed OLE objects into the Information fields of objects. Had to devise a new TK5RichEdit class that registers an OLE handler with the base TRichEdit. The handler is responsible for enumerating and displaying the embedded OLE objects. We're now saying that our K5 information fields are containers for OLE. This move had a great impact on the overall quality of ANNA K5 system as K5 can now assume the role of a document manager.
  2. The RichEdit had to be overridden so that the OLE handler may be destroyed and reconstructed on Create/DestroyWnd.
  3. Another bunch of fixes to the AutoCenter logic of the Schematic Editor. It now also takes into account the zoom factor.
  4. Enhanced the Relation Editor with a pre-compile action that runs the compiler without taking any semantic actions, just highlighting the syntax so that the user may either re-align the text before compiling, or adjust the grammar if necessary.
  5. Enhanced the Relation Editor with a little state machine that correlates the Keep and Okay buttons with the Modified state of the information viewer. The new logic takes into account wether the Keep button was manually or automatically pressed. The reason for this state machine is to disallow the reloading of the relation viewer if its content wasn't yet saved.
  6. Enhanced the Relation Editor wit a safe-delete action that calculates the weight of the structure, prompts the user for confirmation, and then deletes the current node.
  7. Added the Replace action to the Relation Editor. Corrections to the Find logic. The Find /Replace button does a find on the left click and a replace on the right click.

}

CORE {

  1. Information cross references are now special lines rather than expensive relations. This change was required to not pollute the scope with relation fields introduced by the automatic information cross-referencing process. A new XREF class was introduced a s a specialized LINE. Information xref lines are of this new XREF type.
  2. Added a virtual LINE constructor to CAT_ABSTRACT.

}


133 Wednesday, July 14, 2004

BASE {

  1. Moved the CloseEditor to BASE so that everyone can Open and Close an object's local editor.
  2. Caught and fixed the problems regarding the visibility of objects which were due to signed chars that can't be masked.
  3. Unified all the CreateInstance prototypes to take size and init_data as parameters. This was causing problems.
  4. Added the UserName to HOST_OS to get the currently logged-in user.

}

CORE {

  1. Exposed a virtual NODE_Serialize method for loose coupling of GUI to Core.
  2. Stream is using PointerArithmetic to convert pointers to the frontier of aggregated structures from previous versions.
  3. Stream implements two new methods to load and store its children. These are now used to expand stream nodes such as AUTOLOAD.ANA in the GUI.
  4. Made a change to the STREE serializor to not reinsert the loaded node. This is to support the new STREAM::LoadContent.

}

FAXCORE {

  1. Bound the type masters of TFaxFile, TFaxType, and CAT_FAX_CORE..
  2. TFaxType::InsertInstance is new. All instances of fax types are inserted into their type masters right after creation. One can observe the demotion of an object from TFaxObject down to the final specialized type.
  3. TransferString is now a feature of TFaxBase, like it used to be, because it must allocate memory from the same pool where the FaxCore destructors operate.
  4. Fax instances are created with an integral handle so that they are constants rather than selectors.
  5. Promoted the type vector to FAX.CPP and exporting it to the other modules. This move was to reduce and generalize the TFaxType::GetID and TFaxType::GetByID methods.
  6. Added a bitmap for the CAT_FAX_CORE as a Fc in a square. Added a bitmap for TFaxType as a Ft in a square.
  7. The Server is constructed and loaded from the INI file in the startup of the FaxCore rather than in the GUI. The GUI now mainly attaches to an existing construct. The Server destroys on FaxCore shutdown.
  8. Obsoleted the SendCoverSheets of TFaxServer.

}

GUI {

  1. The daemon animation had to be stopped when the About sheet is obscured because accessing the GDI during the handling of the timer may nest and result in contention over the VCL canvases.
  2. Whenever a delayed redraw is issued to the Schematic Editor, the timer is reset by another period. This avoids intermediate redraws, which is what the delayed redraw wanted to avoid in the first place.
  3. Corrected the handling of CreateNewTypeInstance in the Schematic Editor. It needs/should not create a constant handle.
  4. Enhanced the ScopeView to call LoadContent for a STREAM node when expanding one. The stream content is a special binary file incompatible with the others. Enhanced the loader/store to handle this type. The new file extension is ANA.

}

FAXCLIENT {

  1. An attempt to make the inclusion system more regular so that the precompiled header file isn't invalidated.
  2. The Fax Active Directory view is now appending to the FaxCore and gets the Local Fax Server pointer as the first available instance of TFaxServer. The role of the FAD browser was always supposed to be to navigate, the creation aspects were the responsibility of the FaxCore.
  3. Had to add to all Fax Type Editors the FormDestroy method that does an ApplyClick. We were finding that the local editors were not destroyed on close and the changes were not applied.
  4. Caught and fixed a problem reading the string fields of the Server Editor.
  5. The editor of a Fax Proxy persists the port and IP values.
  6. The FaxTypeLib now overloads the OBJECT_BASE::CloseEditor member for obvious reasons.

}


134 Saturday, July 17, 2004

ANSI {

Added the alloca function to malloc.h and with body in alloca.c. This is at the level of stub right now. The MSVC build cannot persist as a consequence.

}

BASE {

  1. CurrentAlias was promoted to a static member of OBJECT_BASE. SetCurrentAlias is obsolete.
  2. Protected the INI methods against a null section string that might develop from persisting unnamed objects.
  3. Modules might need the _argc and _argv global variables. Defining them in HOSTOS.
  4. GetLastError returns errno under GNU.
  5. CloseSocket does a shutdown(s,2) in a (rather futile) attempt to break out of the accept function from another thread.
  6. Bind is trying to set the REUSEADDR option on a socket that fails with EADDRINUSE. The logic isn't effective.
  7. Implemented the SetBlockingMode for Linux as well. Nobody is currently calling it, though. It is part of my attempts to break out of the accept function.
  8. Explained a few errno codes to do with sockets to the debug log.
  9. Implemented ModuleGetFileName to use _argv[0] under Linux.
  10. Corrected the Sleep function to sleep in seconds rather than milliseconds. This was the cause of severe flaws in the server logic as the waiting loops weren't calibrated.
  11. Implemented the UserName function for Linux.

}

CORE {

Another go at consolidating the CreateInstance methods as prompted by the MSVC build that disallows variations of virtual methods in their return type.Added TEditStrm to the MSVC build of the Core.

}

FAXCORE {

  1. The forking of the various threads of the FaxCore are not the responsibility of the command interpreter. The issue of forking is now dealt with in TFaxServer::Start, leaving the command line free of such details.
  2. Added another LOGOUT command to the protocol and reassigned the meaning of QUIT. QUIT is a protocol command that the command shell of the server issues upon the listener thread of the same server to make it break out of the accept function. Unlike under Windows, the behavior of accept under Linux is to hang even when the socket was closed. Under Windows the accept breaks out if the socket becomes invalid which is the case if a foreign party has closed the socket upon which accept is blocked. The behavior under Linux might have something to do with the improper implementation of threads as processes and the cloning of address spaces that makes impossible the proper handling of cancellation points.
  3. Added Printer.CPP to the FaxCore. Implemented the TFaxFifo class. It has now absorbed the printer installation logic that used to inappropriately be handled by the FAD Browser GUI. The Start and Stop of the TFaxFifo deal with the installation of the printer. TFaxFifo isn't doing anything under Linux yet.
  4. The Printer installation logic has had to change to not use any Borland VCL logic, so no more AnsiStrings or TTimers. The printer installer is a Windows bound algorithm that would require a vastly different implementation under Linux if need be. It is possible to install a similar printer FIFO under Linux so that any program that can print also gains faxing capabilities. We would then need an interface to qualify the documents with endpoint addresses, an equivalent of the Windows client.
  5. Protected several methods against unnamed objects. The unnamed objects arise when default objects are constructed.
  6. Changed the way the Server starts and shuts down. The command interpreter first initializes the fax core, then finds the first instance of the server and starts it by invoking the Start method. The command interpreter of the server shuts down the first instance of the server, then shuts-down the server. The scheme might need to be enhanced to either start/stop all instances, or not shutdown the server module unless all server instances were stopped. The command interpreter is simpler.
  7. The design of the Server has changed. A server doesn't own a printer FIFO anymore. The printer is dropped explicitly by the user into a folder of their choice. The printer owned by the server was like a sore thumb in the design. Server has nothing to do with printers now. As a nice consequence, the user can install (for whatever reasons he has) several fax printers into any folder. The printer FIFO in the new concept is a PS file generator for the queue it lives in.
  8. The design of the Server has changed to have two forked threads: the Listener and the Evaluator. The two threads are started and stopped together through fairly delicate system logic that deals with any synchronization aspects.
  9. The Server evaluator period is specified in seconds. The period is in fact a sleeping period where the server is relinquishing time slice a value of 0 means that the server is never sleeping. The move has two reasons: first, the server evaluator needs not run continuously, second the resolution of timestamps of objects is at the level of seconds, so an evaluator that runs more frequently than the seconds might create objects with the same timestamp, which is a potential source of confusion.
  10. The construction of client proxies is currently disabled because the synchronization mechanism is under reconstruction.
  11. Corrections to the Startup/Shutdown of the FaxCore.

}

FAXCLIENT {

  1. The Server has suffered a severe re-design. GUI is much simpler now.
  2. The FAD Browser doesn't deal with running anything so the RUN button is gone. A server is started from its own property sheet. There is no printer install button anymore. One manually installs one or more printers in any folder. The browser toolbar is the simplest thing possible: a button to delete and a palette to insert. The printer test button is still present, but it is ineffective, as the whole business of closure analysis is going to change.
  3. More than one server can now be constructed in a client FAD. They live separate lives and run independently on their own listen/evaluate threads.
  4. Added a refinement on closing the GUI to not have the FAD reselect all the nodes as they get destroyed.
  5. Some changes to the browser drag&drop logic that now permit the creation of clients to the local server. This is a fairly sophisticated concept that would need explaining.
  6. Changed the way the browser logs the tree so that more than one local server may be instantiated in the FAD.
  7. Added the install/uninstall logic and button to the FIFO editor.
  8. Added the server start/stop button and logic for the Server editor. No more printer-related fields for the Server.
  9. The FaxClient editor allows connecting/disconnecting to the server. The button effects a login/logout.

}


135 Monday, July 26, 2004

BASE {

Function vsnprintf in DefaultDebugMessage can hang with multi-threaded users. Had to use a spin-lock around it.

HOST_OS::Send/Recv shouldn't do anything if buffer length is 0.

Reconciled the differences between Linux and Windows in obtaining the symbolic name of the remote server. We're now using HOSTENT and gethostbyaddr which involves DNS and NETBIOS for name resolution.

Enhanced HOST_OS::ApiError with socket related error messages.

}

FAXCORE {

  1. Corrected the TFaxBase destructor to free the object name. This is necessary across multiple memory managers.
  2. A few more protocol commands: GETVERSION, EXPAND, COLLAPSE, LOCKEDIT, UNLOCKEDIT.
  3. Added the flags parameter to the object Transfer methods to specify whether the related objects should also be transferred.
  4. Obsolete methods: PersistObject, PersistField, Set/GetItemState, TFaxObject::Disconnect.
  5. New methods: TFaxObject::CollapseTree, Sync, GetName, GetType, GetClient, Lock/UnlockEditor
  6. New Fields of TFaxObject: RemoteEnd, RemoteType, EditorStation.
  7. New methods of TFaxClient: Login, Logout, GetClient. Obsolete ClentProxy field as replaced by the more general TFaxObject::RemoteEnd.
  8. Obsolete TFaxServer methods: CreateFifo, FifoExists, UncoilFifoToFile, GetErr, ClientProxy
  9. Enhanced TFaxServer::GetSlot with a name parameter. A client computer may connect more than once to the same server under different client names over the same IP:Port pair. This scenario arises on Terminal Servers.
  10. The UserPort on the command line isn't used anymore.
  11. Corrections to the construction of the mutual proxy node.
  12. Extended the destruction of the mutual proxy to have the server first find it, then delete it, to prevent a scenario in which a client sends an invalid client id. Must rethink if this is really necessary.
  13. TFaxObject::Transfer now exchanges the Editor and the EditorStation pointers. These are the only members of TFaxObject that should be transferred so far.
  14. If a local FAD node has a RemoteEnd pointer set to something, then the node is said to be a proxy to another remote node. The path to that node is fully qualified by the following pieces of information:
    1. A client node somewhere in the parenthood of the local FAD node. It is impossible for a proxy to exist outside of a client tree. The IP, Port, and RemoteEnd of that client node will identify the server and its proxy back to this client.
    2. The RemoteEnd of the FAD node identifies a remote FAD node in the context of the client.
    3. The RemoteType is a safety measure.
  15. The new object synchronization method does the following:
    1. Verifies cross-references between the source and target node and consolidates them as follows:

i. we send them a description of what XRefs we have.

ii. we ask them what unsatisfied XRefs should we delete

iii. we get a description of what XRefs they have.

iv. we tell them what lost XRefs they need to delete

v. At this point we've cleared all the mismatching xrefs,

    1. We ask for a difference report on the remote's true FAD nodes.
    2. For any FAD nodes that the remote directory has and we don't we must create proxy FAD nodes.
    3. We send a report of differences with our real FAD nodes back to the remote end.
    4. For each such report entry, the remote end will create a proxy back to our local node.
  1. The new TFaxObject methods to Expand/Collapse use the new client/server crossover scheme. The methods synchronize the two endpoints using the new TFaxObject::Sync method. Further details on the protocol are fond in the report of faxsrv.cpp module.
  2. The new TFaxObject::LockEditor and UnlockEditor methods. A client selects a FAD node and loads its type editor. If the FAD node is a proxy to a remote object, then a virtual local image of the remote object must first be constructed, synchronized, and loaded into the editor. When the editor is closed, the synchronization process must follow the opposite direction from the editor's temporary object back to the remote original, and the temporary object is deleted. Please look for details on LOCK/UNLOCKEDIT for file server.cpp.
  3. Changed the client/server login mechanism.
    1. A client issues the login command followed by the client's name.
    2. A server responds to the Login command by first loading the client name.
    3. The server searches the list of logged-in clients to see if the client with that name is already running. If not, the server creates a proxy client for the client that's logging in.
    4. The server returns the handle to the proxy back to the client.
    5. The server sets the name of the proxy to the name of the client sent at the beginning of the login process.
    6. The server sends back its own name.
    7. The client sets its name to that of the server.
    8. This concludes the login process and the client node now has the name of the server, and the server's mutual client has the original name of the client. In other words, client nodes have swapped names.
  4. Changed the client/server logout mechanism
    1. Client issues the logout command.
    2. Server sends back the name of the proxy client, which is the original name of the client node before login.
    3. Client resets its name to the original.
    4. The server deletes the proxy client node.
  5. New command to EXPAND a server FAD node.
    1. The client sends the remote handle to the server's proxy client. The handle was obtained at login.
    2. The client sends the remote handle to the server object it whishes to expand followed by its type.
    3. The server searches its registration structures for the object of the given type and returns a failure code.
    4. For the found object, the server commences the synchronization process that the client is waiting for.
  6. A new command to collapse a server FAD node.
    1. A sequence similar to that of expanding the FAD.
    2. Once the server has synchronized the node, it will also destroy all nodes with a remote reference. This matter must be carefully considered.
  7. A new command to lock a client's editor onto a server FAD node.
    1. The client sends the remote handle to the proxy client, the pointer and type of the target FAD node.
    2. The server locates the object and sends back the name of the client that currently has the node locked for editing, if one such client exists.
    3. The client aborts the lock attempt if the string isn't null, alerting the user appropriately.
    4. The server will set the EditorStation field of the target FAD node to point to the proxy of the client that's about to lock the node.
    5. The server transfers the node over to the client, and exits.
    6. The client receives the content of the remote object, inside the body of a fake temporary FAD node, and loads that object into the editor for viewing/editing.
  8. A new command to unlock a remote node for editing.
    1. The client sends the remote handle to the proxy client, the pointer and type of the target FAD node.
    2. The server locates the object.
    3. The client transfers the content of the edited object back to the server.
    4. The server coils the incoming data over the FAD node.
    5. The server resets the EditorStation field of the FAD node so that other parties may now lock it.

}

FAXCLIENT {

  1. The FaxCore wasn't shutting down in the monolithic version due to a premature return statement.
  2. Removed the Uncoiler and Listener defunct modules from the client.
  3. Getting the FAD viewer to highlight the remote nodes in blue and yellow.
  4. The FAD viewer issues a CollapseTree on the selected collapsing fax object, so that the FaxCore may synchronize the local FAD object with the corresponding server one.
  5. Established a skipper semaphore around the GUI debug log. We have to skip the debug line if the deadlock isn't resolved in a number (1000) of retries.
  6. The type editors must lock the FAD node before editing it, and unlock it on close. The TFaxObject::LockEditor may return a proxy object for a remote FAD node. The Lock/Unlock calls the editor makes are an opportunity for the FaxCore to transfer the content of the target object and virtualize it locally.

}


136 Thursday, August 05, 2004

BASE {

  1. Thorough rework of the TNullRelation and TMultiRelation constructs to support the popup of the GUI on Notification events. TNullRelation is now a descendent of STREE_BASE. It has a virtual destructor.
  2. OBJECT_BASE has a static AttachedUserInterfaces notification structure and a method to notify all user interfaces of events.
  3. Added the SendRecv method to HOST_OS_COMMS to simplify the user code and keep it symmetric in both directions.
  4. Fixed a problem in Recv to not enter the loop for an invalid socket.
  5. Consolidating a way to create temporary files on all platforms as the new MakeTempFile function.
  6. Added a protection to the INI_TREE against a null section string. In such a case, the fields are considered free-floating.

}

GUI {

  1. Restored the GUI popup mechanism to autofocus on the given object. A TForm_Relation now only has one Relation node to establish a tree. SubRelation is obsolete. TNotifyRelation is a stub. TNullRelation is up-climbing. TTopRelation is new and ends the upward recursions. Any final multi-relation is a descendent of the TTopRelation. TForm_Relation constructs a TNullRelation if no specialized structure is passed through the constructor. The constructors of TNullRelations are auto binding the forms and auto-inserting. A TNulRelation that constructs with no parent will auto insert into the Kernel's AttachedUserInterfaces notification registry.
  2. Directory picker. File open dialogs and controls for TFaxFile descendents.
  3. Added two ConvertTime functions to convert between time_t and TDateTime.

}

FAXSYS {

  1. Reworked the TIFF viewer to allow zooming in and out, zoom to fit to width, zoom to page. Added the zoom factor editor. A factor of 1 corrects the former problem of fax fuzziness due to aliasing. The new full size images are as sharp as Kodak's. For smaller zoom factors, we're going to have to go to grayscale bitmaps for anti-aliasing, which will require special memory management techniques. It's good enough as it is.
  2. Work on the design of the Fax Document Viewer of the GUI type library. Removed some fields that won't ever be used, added the date and time control. Resolved difficult conversions between UNIX time_t and Windows time. Added new persistence and synchronization for the new fields.
  3. Restored the functionality of previewing faxes. The logic depends on the capabilities of the local server and on the connected users. If the local server can't convert a PS to a TIFF, then it looks for connected user sessions. If a user is connected to a server than the associated server is asked to do the conversion as that user and on behalf of the requesting client system. Can view faxes now.
  4. Reworked the functionality of retrieving cover sheets. The new logic is to ask the server at the other end of the first connected user session for the list of coversheet templates.
  5. Major rework of all issues involving the manipulation of temporary files, to keep our directories as clean as possible. Changed the way documents form their names.
  6. Reconstruction of the FAD Evaluator. A queue Evaluator searches its directory for new files and creates nodes for them. Special nodes are created for TIFF documents in the drafts queue. A queue is for drafts only if it is tagged as such from the queue editor. Reworked the FIFO printer to uncoil the PS file, convert it to TIFF and leave it in the Drafts queue. The queue evaluator will come around to create a Fax Document node for the new TIFF file.
  7. The addition of each new Fax Document triggers a notification through the system of registered multi-lateral relations, eventually intercepted by the GUI. Thorough rework of the TNullRelation and TMultiRelation constructs to support the popup of the GUI on fax events. Restored the GUI popup mechanism to autofocus on the latest fax document. Resolved great adversities to do with the threading of a Fax Core that runs in the background without a GUI, issues to do with GUI creation/destruction and attaching/detaching to an already running core. Resolving the final problem of the Tray constructing the GUI in response to a new Fax Document event and auto-focusing the FAD browser onto it.
  8. Reactivated the Closure Test. Promoted the FIFO printer test from GUI to FaxCore as a method of TFaxFifo. The function is invoked from the local editor of the FIFO object. Simplified the tester. Corrected the whole circuit until the tester ends-up popping-up the GUI focused onto the latest fax.
  9. Renamed TFaxUser to TFaxContact, TFaxAddress to TFaxContactList, TFaxClient to TFaxUser, TFaxProxy to TFaxJob. Many roles have changed and settled to what they should have been.
  10. Changed the login mechanism to answer a variety of issues such as the first login that creates a new user, logging in as administrator. The client machine configures a user node with a name and password, then logs in. If the password matches the server's then the client is logging in as administrator for that server. A user of that name is either matched or created. If the user exists, than the password supplied from the client must either match the user's or match the server's. If login is successful, the client FAD user node is synchronized with the corresponding server node.
  11. A server FAD node corresponding to a user logging in as administrator differs from the normal user nodes in that the server adds to the user FAD node a symbolic link to itself. This is all an administrator needs to start browsing the server from its top.
  12. A quick and nice enhancement: every FAD node that starts with "Shared " will be vlinked to all users. This permits multiple users to share certain objects in the FAD directory such as the Inbox or Contacts may be.
  13. We need to open the remote FAD nodes into the editors of the local machine. This is a sophisticated matter resolved as follows: Each editor locks onto its object before loading it into its viewers. The editor assumes that the object pointer may change as a result of a Lock operation. Locking will create a proxy image of the remote node into the address space of the machine which locks the object. Unlocking will synchronize the remote node with the image of the proxy. Enhancing the object lock/unlock protocol of the FaxCore. Corrected the whole GUI+Core until all remote objects can be edited.
  14. A user logged into a server as administrator can view/edit the server node. Corrections and enhancements to allow the server to be edited. One may now change the server's password in this manner. Adding to Status variables to mark the state of a server's threads. A server's Listener and Evaluator threads may be one of STOPPED, STARTING, RUNNING, PAUSED, or STOPPING. The server editor takes the status variable into account when conditioning the Start/Stop button. A remote machine may now be stopped from the local editor. Naturally, only the administrator has browsing capabilities over the server and can stop it.
  15. TFaxJob is new. It reuses the design space of the defunct TFaxProxy class. It started as a simple core object. Its GUI type library viewer has all the fields of a HylaFAX state file arranged as nicely as possible.
  16. Re-implemented the Fax Send mechanism as follows: The server evaluator looks to promote everything that's in the Drafts over to the Sendbox if such a queue is present in the local FAD. A Windows client machine is configured without its own Sendbox unless a Windows version of the sendfax tool is installed here. A connected user node may be expanded. If the real user node residing on the remote server has a Sendbox then the local FAD evaluator will see its proxy image and attempt to promote the local draft documents over to the remote Sendbox. Enhanced the Login scheme to automatically add the Sendbox and Inbox for each user. All drafts that are fully qualified may be marked for transmission/promotion. The user decides when to send them by pressing the SEND button on the Fax Document viewer. The local evaluator will transact with the server's listener each document so marked. The local document is then deleted as the document continues to live in the remote sendbox. During the transfer, all the contacts are also transferred, but they will be cloned to the server. This is a complex issue. A notification is raised on the receiving server for each new document received. Testing the scheme to satisfaction.
  17. The sendfax logic is way out of sync now. Changes to the Exec routine to use a Linux native fork and clone the process ID. New logic to do with temporary files. Another parameter to get HylaFAX to preserve a TAG id back to our document. HylaFAX reports the new documents we send one per each contact. Need a way to relate HylaFAX's sendq/qxx files back to our document.
  18. Enhanced the server to added the docq, sendq, recvq, and doneq, TFaxQueues with directories mapped over HylaFAX. Enhanced the queue evaluator to create generic file nodes for all new files in a folder, and create special document nodes for TIFFs into the Drafts queue, of TFaxJob nodes for qxxx files in the HylaFAX watched directories. The queue evaluator cross-references Job files with corresponding TIF files. Testing. A client machine can now browse the HylaFAX directory structure and expand a Job to see its related files, and expand a file to see its related jobs.
  19. TFaxJob has all the fields defined by a HylaFAX qXXX state file. Added the Evaluate method to TFaxJob to reload the content from the HylaFAX state file. The server's evaluator will keep the Job nodes synchronized with the q files.
  20. Enhanced the SEND protocol: the server invokes sendfax to create the state files, then it cross-references all Job nodes with the corresponding contacts of the fax document. This way a client machine may browse into the contacts of a sending document and view its job status.
  21. Enhanced the TFaxJob viewer with a refresh timer to update every 10 seconds so that the user may poll the status of a document in progress. Corrected some contention problems between the GUI thread and the evaluator thread of the client machine. Testing the send processes to satisfaction.
  22. Thorough consideration of error reporting from the server to avoid blocking on responses that will never come.

The fax system now sends and receives faxes.

}

FAXCORE{

  1. New protocol commands: XCHGOBJ ,DOCSEND, CONVERT, GETJOBSTATUS, INSERT, ERASE, FIND,
  2. TFaxBase::set_Name is in charge with deleting the old name. The destructor sets the name to 0, to prevent the allocator of the lower level to attempt to free a block that doesn't belong to the module.
  3. InitializeFaxCore now saves the startup directory into the InstallPath global string.
  4. TFaxServer is a TFaxUser descendent. Added the TFaxServer::Evaluate and Transfer methods.
  5. Caught and fixed a problem in counting the contacts in a recursive list.
  6. Implemented the TFaxQueue::SyncDirectory for both windows and linux using findfirst/findnext or scandir functions. Under Linux, the ScanDir creates documents.
  7. TFaxObject::Erase triggers the server ERASE protocol for proxies to remote nodes.
  8. TFaxObject::Evaluate simply recurses the children.
  9. TFaxObject::Transfer needs to also exchange the Associates parent to allow the client to browse symbolic FAD links as normal remote nodes.
  10. The root FAD node, typically the server, returns the InstallPath+Name in a GetPath call.
  11. Corrections to ExpanDree.
  12. Added another level of sophistication to TFaxObject::Sync to do with the transfer of associated nodes.
  13. UnLockEditor only starts the transfer if the object isn't dirty. A speed improvement.
  14. Better template data.

}

FAXCLIENT{

  1. Added the printer.cpp and the new DirPick.cpp files to the monolith build.
  2. Caught a problem in that no CloseEditor procedure was installed. Changed the CloseEditor function to be issued by the multi-relation. Code moved away from WinMain and into FaxEdits.cpp.
  3. TMultiRelation::SetData finds and expands a node in the tree viewer.
  4. Caught and fixed a problem in closing the previous editor. It had to be destroyed rather than closed.
  5. Any notification caught at the level of Tray, is indicative of the absence of a GUI form. and that one must be created. Creating the GUI form must happen on the Tray's main thread, so a set of variables are configured for the Tray, and the tray's timer is started. The timer will construct the GUI and focus the FAD browser.
  6. Added a button on the FAD browser's toolbar to rename the selected node.
  7. Added the FindExpand logic to find and focus a given object into the GUI tree.
  8. Moved the integrity tester from GUI to the FaxCore as a method of TFaxFifo.
  9. The biggest problem in closing the GUI form while the server is connected and running was that the WinSock library was shutdown. Corrected.
  10. Simplified the FAD viewer's drag&drop logic to either move the node under the new target or create a symbolic link when ALT is pressed.
  11. The Erase function isn't loading the INI file to remove the section, because the INI is re-written on close.
  12. Reworked some bitmaps to reflect the new names and roles. Large User, small user, user2. Bitmaps for the zoom toolbar. Added a new bitmap for the rename button.
  13. Enhanced the TFaxFile viewer with file loading/storing capabilities. Added the browser line and button. Added the save confirmation button when modified.
  14. Added the browser button and the directory picker to the TFaxQueue viewer.
  15. Added a refresh timer to the TFaxServer viewer to poll for the actual start/stop of the local or remote server.
  16. Added the masked password editor and confirmation box for TFaxServer and TFaxUser viewers.
  17. Logic to enter and verify the passwords.
  1. Reworked the TFaxDocument viewer to allow zooming in and out, zoom to fit to width, zoom to page. Added the zoom factor editor. A factor of 1 corrects the former problem of fax fuzziness due to aliasing. The new full size images are as sharp as Kodak's. For smaller zoom factors, we're going to have to go to grayscale bitmaps for anti-aliasing, which will require special memory management techniques. It's good enough as it is. The zoom factor is a global variable so it applies to viewing all fax documents. A new look for the SEND button, which gets replaced by the HOLD button. Added the date and time picking to the TFaxDocument viewer. Time conversions between ANSI and Windows. Persisting the retries and redails.
  1. Added a status bar to the TFaxEdit ancestor of all type library viewers. Added the SetStatus method to the base class.

}


137 Saturday, September 04, 2004

BASE {

  1. Added the ComputerName method to HOST_OS to return the NETBIOS name of the local machine. Under Windows this is done with a call to GetComputerName, under Linux it is done with a call to gethostbyaddr using the null ip 127.0.0.1.
  2. Added the ResLoadString method to HOST_OS to return a specified resource string.
  3. Added the HOST_OS::ExtractResourceToFile method that now permits storing files inside the resources of an EXE.
  4. Added the direction flag to the INI_FILE constructor. This is to prevent unnecessary overwrite of the INI file after reading it, or the unnecessary read of the ini file before over-writing it. These are subtle matters prompted by the new requirement to rebase the structures before storing and after loading.
  5. The new INI file logic stores objects as [Name@address]. The storing objects are renamed to Name@address before storing and renamed back to their original Name upon the destruction of the INI file manager. PersistPObj effects a silent rename to the Name@address scheme prior to storing the object. The INI_SECTION destructor rename the linked object back to its natural name.
  6. The storing of an object's children is now done using the qualified Name@address scheme. OBJECT_BASE::Persist builds qualified children vectors.
  7. Changed the storing of the tree aspects to allow for inter-level tree insertions.
  8. Promoted the object renaming logic to OBJECT_BASE for more consistent memory management.

}

CORE {

  1. The OBJECT destructor had to be virtual. Added another VMT identifier to EVmt.
  2. Modeled GetFlatID and GetByFlatID methods.
  3. The OBJECT::DONE destructor uses set_Name(0) to release the name space.
  4. OBJECT::set_Name now deferes to the inherited OBJECT_BASE::set_Name.

}

FAXCORE {

  1. An unnamed TFaxUser structure is now set to have the name of the currently logged in user.
  2. The name of the fax server is now taken from the NETBIOS name of the local computer.
  3. The IP and Port of the windows client is now read from the resource strings 2 and 3.
  4. The default command line of the Windows version is now read from resource string 1.
  5. No need for TREE::set_Name. the role was promoted to OBJECT_BASE.
  6. The NT printer drivers were added to the resources of the Windows version, to be extracted and installed.

}

The logic of overloading the Open/CloseEditor had to move out of FaxTrayMulti and into FaxTrayMainMonolith.

Enhanced the BC502 console application to load the FaxCoreLib module, then extract the faxcore_main entry point and call it.


138 Wednesday, September 22, 2004

Scope Browser {

  1. Added the Keep button to the Scope Browser to prevent it from refocusing. Refocus is often undesirable. The auto collapse may inadvertently save the AUTOLOAD morphic file.
  2. Added support for loading the topology files of the Australian Railway System of Nodes, Lines, and BSL.
  3. Added support for loading the NICOR style of nodal database from Dan Anastasiu.
  4. The Morph category has a new method to import a file such as NICOR under a specified format.
  1. Drag and drop was enhanced to respond to sources such as the master tool buttons of the Schematic Editor, which represent the interlocked selection on the schematic restricted to the type identified by the button.

}

K5CORE {

  1. Added the Persist vector of methods to serialize to an INI file.
  2. Found a major problem in the Solid Frontier Serializor when fixing the pointers to containers.
  3. Found a serialization problem and had to alias CLSID_USER_TYPE to CLSID_NODE.
  4. AreaMatch function used by CAT_ABSTRACT::Import.

}


139 Friday, September 24, 2004

{

  1. Worked on a help file for the Schematic Editor
  2. Schematic Editor inherits from TForm_Relation now.
  3. New Line static flags 8 and 9. New grouping of static flags
  1. Added ofHideLabels and ofDisablePathFinder to NODE.
  2. Enabled the Path Finder algorithm. Controlled by a global NODE option. This mechanism highlights cycles in the wires of a schematic, an important result of the visual system. Path finder starts with NODE/LINE::LockPath and works incrementally from the last selected node to the next, which is why the previous selection needed to be cycled through the core during pick-correlations.
  3. DoSerialize takes another object pointer parameter to pass it as the selection argument to SerializePObj, to support the new path-finder algorithm.
  4. Enhanced the /Drag&Drop system to allow typed dragging of the selection from the master tool buttons. We're now using this filtered drag&drop logic between Scope and Schematic.
  5. Finally resolved the ugly matter of popup menus in multi-threaded GUIs. It's a problem of cardinality in VCL's popup-list.
  6. The NODE master button has a drop-down menu with the tiling algorithms.
  7. Enhanced/Corrected the SelectCone logic to factorize the editors. A cone-selection, however is unlikely to find anything larger than a common TREE aspect.
  8. Fixed some crash conditions in the Create/DeleteInstance.
  9. Changing a global option stops the refresh timer and forces a redraw.
  10. Enhanced the bitmap painter to include the highlighted selection. This was very important to make our technical drawings.
  11. The Schematic Editor now responds to morphism requests issued by dropping a type master over the master buttons of the tool bar. The effect of such a drag and drop is that all selected nodes of the type identified by the master button will be morphed into the new dropped type. The operation may either increase the size of the structure or shrink it, in which case it would be with loss of data.
  12. Added the option of hiding the lines of the schematic, and one to hide the labels of the nodes.
  13. When showing selectors the wire drawing algorithm also changes to follow. We can now either draw wires to dialer pin-outs or normal nodal schematics. (An important result)

}

Corrected the About Form to not replicate the version every time the form was opened.


140 Saturday, September 25, 2004

GAIUS5 {

  1. Added support for hidden RTF references to related concepts. The Set/GetHiddenReference are animating the RTF control into inserting hidden text. We're wiring into the document a K5REF_<object pointer> string.
  2. The Text generator wires a hidden reference before every sub-section.
  3. LoadObject always generates the full text now.
  4. Clicking onto an RTF control needs to be intercepted on mouse-up to not disturb the control's default mouse processing.
  5. Added the ability to click on a sub chapter and refocus the concept view.
  6. Clicking on the top of the editor will take us up one level in the concept tree.

}

GUIFRAME {

  1. Enhanced the logger to stop scrolling if the cursor isn't at the very end.
  2. The logger belongs to the last GUI Form that was launched. Care had to be taken to rotate the owner of the log when GUI forms were closed.
  3. The problem of menu popup in our multi-threaded GUIs is due to the PopupList->Window of VCL which belongs to the main thread. Our GUI threads must re-create it per thread, and revert to the original for the tray.
  4. Enhanced the application's help-file resolution. The active GUI frame is checked for a help file, then the application.
  5. The activation of the GUI frame switches the context of PopupList->Window of VCL.
  6. Hiding the debug log now disables the logging system altogether. This is sometimes safer.

}


141 Sunday, September 26, 2004

NAVIGATOR:

  1. Added the FaxDirView module to ANNA K5. This is an alternative browser for distributed objects. It may load the FaxCore which in turn loads the FaxTypeLib editors. The Navigator can build without FaxDirView from a define macro.
  2. Added support to reload the interface by double-clicking on an instance listed in the Type Viewer.
  3. We had lost the ability to bypass the DxD driver and paint the schematic onto the window device context. Added another option flag to DxD to skip the construction of DxD.

BASE: A fix to dumping the INI configuration logic to console.


142 Monday, September 27, 2004

BASE {

  1. Added the stub LICENSE class to BASE. It just has a data string and a verify method. Descends from DEBUG_BASE.
  2. Bool IsXRef. An object is an XRef if the object and its parent relate at different tree aspects.
  3. GetStates is a new virtual method that the core objects override and return the following flags: gsLocked, gsReference, gsRemote, gsModified.
  4. Methods GetName and GetAlias now return the states defined by EGetStates.
  5. Relate is a new OBJECT_BASE method to abstract the link creation logic.

}

Scope Browser {

  1. Drag and drop It is now calling the new object base Relate method with the alt modifier parameter. System Core software will decide how to resolve the relationship.
  2. Enhanced the drawing of the tree with a discrimination of nodes for reference, modified, locked, and remote scenarios, as described for each node by the new OBJECT_BASE::GetStates method. The look of the tree has changed.

}

K5CORE {

  1. OBJECT has the new Morph method to support structural morphism of objects from one type to another. The method takes in an exchange buffer for initializers and shrink data.
  2. The Semaphore field of an object had to be initialized to 0.
  3. THierachy had to implement get_Supplier like a Relation and return itself.

}


143 Tuesday, September 28, 2004

ANSI_K5: Added extern char *fgets(char *string,int n,FILE *fp) to STDIO.H


144 Saturday, October 02, 2004

BASE {

  1. Homogenized the inter-module memory allocation. Include the local_mem.h header file to rename the standard malloc, calloc, free, realloc, and strdup to the local_malloc, ... etc unique functions of the base. Include host_mem.h to revert.
  2. The local_malloc functions will route the call through the MemoryAllocatorFPtrs vector which is installable by core modules.
  3. Moved the definitions of _argv and _argc out of the base module.
  4. New methods HOST_OS_RT:: GetOSSerialNumber and GetUniqueID.
  5. Added a counter of GDI resources allocated to hunt a leakage.
  6. Chasing a hang in TCP/IP. The scheme of always spinning around send and recv regardless of errors isn't right. Preparing HOST_OS_COMMS SendBuf, RecvBuf, RecvBufLead, SendFile, RecvFile, TransferString to return status. Returning the number of successful bytes transferred in Send and Recv. Logic to sense the underflow in the other string oriented transfer methods.
  7. Caught a problem in TransferString when receiving a 0-sized string was not zeroing the argument pointer.
  8. Corrected the byte ordering problem in binding and connecting to a port using htons. Also wrapping the port to short.
  9. A fix to the ResLoadString stub for GNU.
  10. Caught and fixed an imbalance of GDI SelectObject calls in building bitmaps for the tree views.
  11. Pulling some stunts to MakeTempFile under the MSDEV tool chain that lacks creattemp or mkstemp.

}

K5CORE:

  1. The Morphism Core replaces the MemoryAllocatorFPtrs upon startup and restores them to the original on exit. Memory management calls are routed through the Morphic Memory Manager.
  2. CanvasStream was leaking fonts. Also gave up the attempt to prevent pen and brush reload, as futile.

145 Saturday, October 09, 2004

Components {

  1. Added the FilterType to TTreeViewWnd.
  2. Enhanced TTreeNodeExt to not create duplicate proxies.
  3. Added the TTreeNodeExt::FindExpand algorithm that recurses up the object's parenthood until it finds a root that is logged in the tree view, than expands the view upon backtracking. Method returns the proxy of the object.

}

Scope Browser {

Constructed the ORDER and FILTER bitmaps

}


146 Monday, October 11, 2004

BASE {

  1. A new class TSpinlock that spins around a static flag if the current thread isn't the owner. Should become a template. Need it to protect the memory manager.
  2. We've been getting nowhere with these sockets. Changing so that Send and Recv call a common NbSendRecv which spins for errors and exits on success or on a blocking warning. Implemented underflow tests and status returns from all COMMS transfer functions.
  3. Using the mode argument to SetBlockingMode in the ioctl call.
  4. Under Windows GetPeerInfo is much too slow through all those DNS layers and what not. Disabling the name info.
  5. Improved ApiError to not display the error messages unless a title string is passed in.

}

K5CORE {

  1. Needed to protect the Memory Manager with a spin-lock using a dedicated class.
  2. Speed-up the Morphic Memory Manager by rotating the Allocator's Children pointer to be the largest known block.

147 Don't know when ???????

BASE {

  1. The non-blocking loop for socket IO is extremely delicate. We may end-up here from a GUI thread. Blocking would be catastrophic. Looping around the same. Sleeping won't do. The only thing to do is the re-enter the messaging loop. This is another delicate story once again, because then the sockets may be re-entered, and so on. But we need to let the GUI messages through.
  2. Simplified the failure returns from the socket related functions, as well as the tests. We're not interested in the successful run-length, just boolean. Send/RecvFile are also returning a success state.
  3. ApiError is calling strerror now, rather than us trying to switch the errno code.

}


148 November Don't know when ???????

BASE {

  1. Added the xrename(const char*oldpath,const char*newpath) function that does copy and delete if the source and destination cross a device boundary.
  2. Added the MakeDir function to level-out the differences between Borland, MSVC, and GCC mkdir functions.

}


149 Tuesday, October 19, 2004

GUI {

  1. Designed and generated Config.h/cpp. The former arrangement of TNullRelation and TTopRelation had to be enhanced so to get rid of the double trees. A new messaging system was designed and implemented to strap VCL's component parenthood.
  2. TTopMultiRelation is new and caps the upward recursions. It also generalizes the endpoint containers, so that descendent applications are slimmer.
  3. We needed an abstract notification to close the editors attached to an object.

}

Navigator {

  1. The TTopRelation implements the generic correlations.
  2. The Enterprise Navigator overrides the Focus and SetData methods to implement the ScopeView - Schematic Editor correlation function.
  3. Reload is a correlation between ScopeView, SchematicEditor, LawView, TypeView, and InfoView.
  4. Reload with a level of -1 is generally issued by scope browsers when leaving a node before focusing another. This is an opportunity for the application to close the editors of the outgoing node.
  5. The refresh notification is for the Schematic Editor.
  6. GetDragSource is invoked by the Schematic Editor or by the Type Editors, and returns the currently selected node. The FaxDirView has precedence over ScopeBrowser
  7. Initializing the WSA must be done once on the main thread.
  8. Initializing the core to GUI notification timer. This timer also toggles the menu to load/unload a core module.

}


150 Tuesday, October 20, 2004

K5CORE: GetTlsData may return a null pointer in the absence of TLS. STREAM::get_Current has to check.


151 Wednesday, October 27, 2004

BASE {

  1. TNotifyRelation::ENotifications, and Close method
  2. OBJECT_BASE taTypeInfo was promoted.
  3. Method GetID is a new object virtual defaulting to return -1.
  4. Added the do_fork specifier to DoNotifyUIs. The function is now forking.
  5. DoReturn is not calling pthread_exit under Linux, and rather returns to the kernel.

}

Scope Browser {

  1. Scope View inherits from TForm_Relation now.
  2. Changing the selection should make an attempt to close the editors of the exiting node. The Relation understands the -1 info parameter as a request to close the editors of the data node.

}


152 Thursday, October 28, 2004

Navigator: changes to the correlations.


153 Wednesday, November 03, 2004

GUIFRAME {

  1. Fixes to the rebuilding of the main menu when a new form is docked to one of the frame's docking sites.
  2. The debug log is a real-time workhorse and the convergence point for all threads. There are delicate problems surrounding a multi-threaded logger: it cannot be allowed to block but it cannot be allowed to skip messages either. It can recurse as a consequence of not blocking. Had to fix a whole heap of problems with it.

}

CONFIG {

  1. TTopRelation::Activate was corrected to re-link the tree of relations to reflect the activation of docked forms. VCL is screwy as far as this goes. We need this mechanism to determine which help module to load, depending on what form is active. This isn't trivial with our nested docking frames.

}

K5CORE main program:

  1. Loading and unloading the FaxCore module. Getting the export points to the main console and calling them.
  2. Added the faxsys_ls listing test.
  3. Creating and closing a socket to initialize WSA in the main thread.

154 Sunday, November 14, 2004

BASE: Changed OBJECT_BASE::set_Name to not do reallocation when the new name is the same pointer as the old name. We can now use this set_Name function to notify the GUI proxies by doing p->set_Name(p->Name). This new way resolves the matter of renaming objects using the in-place tree editor.

Fixed a destruction problem with the docked forms. We're not refocusing the tab control when the GUI is destroying.


155 Tuesday, November 16, 2004

BASE {

  1. A correction to the object persistence algorithm: must chain all the tree aspects of an object even when the object has already been constructed and loaded along another tree hierarchy.
  2. Finally resolved the problem of reusing IP:Port pairs. We're constucting all sockets with the SO_REUSEADDR option, like FTP.
  3. A protection against double entry through the thread detach code that wants to de-allocate TLS.
  4. DestroyThread was going through BASE_LibEntry twice and crashed. This was the prime reason why the server could not reboot.

}


156 Friday, December 03, 2004

BASE {

A fix to our scandir so that it doesn't crash on the second pass over the file list when some files may have been moved away or deleted by async processes

}


157 Monday, November 29, 2004

BASE {

  1. CloseHandle is implemented for Linux as a simple close.
  2. Reporting the API error when MakeTempFile fails.
  3. joinargs concatenates an argv vector back into a command line string.
  4. Implemented the pipe function for Windows as a call to CreatePipe with inheritable handles.
  5. Two new functions Alloc and FreeConsole used by the new IExec function. IExec spawns other programs with redirected handles for STDOUT and STDERR. Gui programs have to first construct a console before they can intercept the output of a spawned program. Waiting for a spawned executable had to be done with a cwait call rather than the WaitForSingleObject.
  6. Implemented the recursive version of MakeDir. RMakeDir constructs the entire path when the upper nodes are absent.

}

GUI {

1. GuiFrame was caught resolving the help file name to the current directory, a mistake. Help files are relative to the EXE file.

2. A new scheme of safely closing the object editors, in the real-time conditions of the core.

3. A new method to TForm_Relation courteously called PleaseDoThatClose. This is a request to close and destroy the form in a way such that the core wouldn't attempt to over-close it. The function works in conjunction with CloseEditor which is what people must actually call. PleaseDoThatClose also checks the application's termination state right after the possibly lengthy object unlock operation, because nothing in the GUI is safe after program was notified for termination.

4. The generic TForm_Relation implements the final PleaseDoThatClose doing a Release instead of a Close. Subtle changes to the destruction of TForm_Relation. Protected the Close method of the TNullRelation, which isn't called much these days all in favor of the base CloseEditor.

5. The core calls CloseEditor now. This function ensures the safety of closing the editor in the following steps:

i. if the application is terminating, then any further GUI operation must be avoided;

ii. if the source object doesn't have an editor attached, then the function exits for lack of point.

iii. if the multi-relation hasn't got a container it means that it (the relation) is deleting, and we mustn't go on destroying this editor.

iv. if the VCL form is flagged as Destroying, then we are coming through this code for a second time because the core has not yet detected that an editor is no longer attached to the source FAD object and keeps calling the upper GUI to close. The core clears the editor field as the last possible step taken by the object unlocking algorithm.

v. If the VCL form is now found to be loading, we are in a scenario of closing the editor right after its opening began. In this case we need to only Close the VCL side of things and not attempt the object unlock because the lock hasn't yet been done.

6. Clicking the mouse on the various fields of an editor had to re-align the help system through the Active vector of the multi-relations. Editors must call Click() on as many controls as possible. I actually disagree with the Windows scheme of mouse clicks being focused events, which is the seed of evil for this scheme.

7. We found a dangerous and useless duplication in the browser where DestroyWnd through some VCL dodgy thinking was re-creating the window handle forcing us to also re-log the FAD tree, making us believe that we have more nodes to destroy.

8. Chased the various problems of application shutting down for days. Eventually clued onto the disastrous effects of not registering the type editors with the multi-relation upon opening them.

9. The tray now waits for the application to terminate before exiting to VCL's destructor. It sets the new TTreeNodeExt::TotalCount field to 0 so that the core knows not to trust any GUI tree viewers as of now on.

10. It became necessary to measure the total number of tree nodes in a process and use that count in closure analysis. A potential problem was then caught in the destructor of TTreeNodeExt not calling the OnFreeObject notification.

11. Changed the FindExpand logic to not rely upon the tree view's Items->Count. Certain threads may delete nodes from the tree while others parse them to find something. The parsing thread would overrun.

12. Added several new protections over the extended tree view and its nodes.

13. Added a protection to TTopRelation::Activate to not assign an invisible form to the ActiveControl of a form.

}


158 Thursday, December 16, 2004

GUI {

  1. The local menu of the Schematic Editor now calls a generic vector called DefaultAction. The Core overrides this method to implement Start/Stop and Login/Logout.
  2. CAT_MORPH->User.Data isn't available anymore. This logic is being re-evaluated.

}

BASE {

  1. LICENSE class moved into base.h/cpp.
  2. BASE has a License instance now for a product called "Enterprise Micro Kernel".
  3. Added the VerifyLicense vector to OBJECT_BASE and to some of its major descendents.
  4. Added the Crypt function to HOST_OS.
  5. Added the HOST_OS::InputBox and MsgBox functions. InputBox is responsible with popping-up a message and allowing a user input prompt.
  6. Added the Transfer vector to OBJECT_BASE and descendents.
  7. Added the SetStatus method to OBJECT_BASE.
  8. The BASE module now takes note of its hInstance.

}

LICENSE {

  1. Implemented the HOST_OS_RT::GetOSSerialNumber function to read the ProductID from the registry and mangle it destructively into a smaller __int64.
  2. Added the HOST_OS_RT::GetLicenseKey and SetLicenseKey functions that operate on the registry.
  3. Designed the LICENSE class to have: Date, LifeSpan, NumberOfRuns, ProductID, CompanyID. The Data field was older. GetLicenseKey is new.
  4. The current algorithm of LICENSE::Verify is:
    1. Return true if NumberOfRuns is greater than 0, or return false if the NumberOfRuns is negative.
    2. Get the challenge key as the current license string from the registry based on CompanyID and ProductID.
    3. Construct the ActivationKey as the actual key encrypted with the OS serial number, shifted into the ASCII HEX readable range.
    4. Calculate the actual license data encrypted with the OS ProductID,
    5. Encrypt the actual license with the ProductID and the CompanyID keys, then shift the value into the ASCII range for readability
    6. Compare the actual with the challenge keys and set the NumberOfRuns to 1 on identity, then return true.
    7. On mismatch, build an InputBox to prompt the user to replace the Activation Key calculated so far with the one supplied by us.
    8. If the user changes the ActivationKey then this new value goes into the registry and the verification process starts all over again.
    9. If the user knows our development password, then the calculated Activation Key is stored and the process restarted to succeed.

}

CORE {

  1. The local menu of the Schematic Editor now calls a generic vector called DefaultAction. The Core overrides this method to implement Start/Stop and Login/Logout.
  2. The MemAlloc, MemFree, MemRealloc, Reclaim, Grow methods of ALLOCATOR were promoted the TYPE. Any type master is now a memory manager. GetStartAddress and GetEndAddress are now used to ultimately map into an address space.
  3. ALLOCATOR is now obsolete. Its position was taken by the new class USER.
  4. Added the FindInstance method to TYPE.
  5. USER is a type master with IP:Port pairs and Status data fields. It has the following methods: Listener, ConnectCommand, Disconnect, CommandDispatch, Login, Start, Stop, Login, Logout, and DefaultAction
  6. Category ABSTRACT now descends from USER rather than ALLOCATOR and instantiates Systems.
  7. Improved the default constructors to also insert any contained objects into the constructing object.
  8. The constructors may now return NULL if they can't validate the license. All descendents must check the result.
  9. Instances of ABSTRACT get the name of the computer as reported by the host operating system.
  10. Any instance of USER gets the name of the current user as reported by the login manager of the host OS.
  11. Implemented the Listener thread of the USER class. Implemented the User login procedure to the level of a simple connect. Implemented the network command dispatcher.
  12. CAT_USER contained by MORPH is almost obsolete. Its Data field is now dummy.
  13. MORPH now descends from TYPE instead of ABSTRACT. MORPH contains the ABSTRACT category instead. In effect we've gone back to an older arrangement.
  14. The Morphism Core is licensed. An attempt to mount the module fails if a license isn't found.
  15. MORPH ends the binding of GetStart/EndAddress into the memory space.

}


159 Sunday, February 27, 2005

GUI {

TCGauge had to change to descend from a windowed control (TPaintBox) because we want to double-click on the indicator to cancel the thread. Added the UserData field to the control to be used as link to the monitored stream.

}

GAIUS5{

Added another button to Gaius5 to turn relations into XRefs. Turning a relation into an XRef must merge the names and info fields of field and connector.

Added logic to allow editing of cable names in the IO browsers.

Added the ShowName and ShowInfo check boxes which influence the code generator. I'm using it after a HTML/XML parse to generate only the effective text of a WEB page.

Added another checkbox to tell the dictionary browser to generate cable information down to a specified depth level

Added another flag to instruct the browser to not generate hyperlinks. We need this to export clean text.

Added another flag to instruct the browser to construct Relations or XRefs when nodes are dropped over the IO panels.

Tool bar is too large and was promoted to the top frame.

}

Scope Browser {

Added a checkbox to tell the loader to use cached streams. User is in charge of that decision depending on how large a file he wants to load. Added a control to configure the stack size of the serializor. Added the new XML import vector.

Reworked the progress indicator to be a NotifyRelation linked to both the monitored stream and the TCGauge control.

Corrected a problem with drag&drop between two instances of the Scope Browser.

}

BASE {

Found a major piece of trouble in HOST_OS_RT::FileAccess when the IO is pending. The seek pointer wasn't incremented. We need to get off this absolute seek pointer in the future and replicate it for each stream instance. Found and fixed a crash in scandir.

Added the Get and SetStackLimit functions that operate on TLS. SetStackLimit is called internally by OBJECT_BASE::Fork.

OBJECT_BASE::ExpandTree logs the tree in the reverse children order.

OBJECT_BASE::Fork promotes the stacksize argument. CreateThread takes the stacksize argument.

OBJECT_BASE::CreateThread now takes the stacksize argument.

Had to expose the CurrentThread method at the OBJECT_BASE level for GUI linking.

}

CORE {

Added the stacksize argument to the STRAM::INIT for threads. STREAMS have gained the ProgressIndicator notify relation.

STREAM::GetToken had a limit of 1000 chars and was crashing for XML loader who wants everything betwen two tags.

Constructing a stream with a cache wrongly assumed that the allocator would always find cache memory.

OBJECT_BASE::CreateThread now takes the stacksize argument. Same as ABSTRACT::CTR_THREAD.

ABSTRACT::Import takes another flags argument used internally by Import to manage its recursion. Added the XML importer.

The Import method is now recursive over the directory structure, looking for files that match a given mask.

The new XML loader is the LoadXML function that parses a given looking for tags and building a tree as well as a schema. It uses the following new sub-functions: GetToken, JustBlanks, StrCmp, Substitute, StartSection, CollectInfoLen, CollectInfo, and the foreign ExtractDictionary function.

ExtractDictionary is a new and sophisticated encoder/compressor that wraps a given text over an existing dictionary using SetInfo. It adds the missing words, adds the new inter-word links, then encodes the input string based on the path taken through the dictionary from one word to the next. The dictionary logic logs its activity and its profile information. One retrieves/decompresses the info text with GetInfo.

OBJECT::Error and Progress were promoted to OBJECT_BASE for linking/overriding by GUIs.

TREE::FindCreate changed with the separators vector, possibly affecting the Rose loader.

OBJECT::GetInformation and SetInformation are new and meant to work with a dictionary.

Sped-up the storing of Enterprise XRefs by caching the interlock ID into the STREE::Vector. The loader side is still a non-linear process that searches the interlock list.

Re-worked the STREAM::StackSize and progress indication logic using the new GetStackLimit function that uses TLS.

Added NODE::get_FirstChild and GetPrev functions for reverse circulation. ExpandTree logs the tree in the reverse order, first node at the top, last child at the bottom.

Found and fixed a non-linear algorithm in LINE::Find(from,to,...).

}


160 Friday, December 24, 2004

Added a command to the command-line interpreter to invoke the DefaultAction of the current node.

Initializing the sockets library consumes about 100 bytes of heap that we can't get back because we don't control the unloading of the sockets module.Moved the creation of the first socket into the BASE module loader.

TLS memory is now allocated off the foreign heap

Added the Default Command to the command line interpreter.

Re-enabled the post-shutdown integrity test.

MORPH::CAT_USER was renamed to USER_GROUP

VerifyLicense returns boolean instead of the platform dependent bool. LICENSE::Verify returns true under Linux.

For the TIGHT_MEMORY_CONTAINMENT scheme to work, all containers need to bind their sub-structures before constructing them, otherwise the Morphic Memory Manager cannot finish the call.

Work on SetTypeMaster to cope with tightly contained memory blocks.

We are now instantiating OBJECTS as part of the workings of the USER class which is a memory manager constructing handles. Had to update the static virtual functions to also expect OBJECT instances.

Started the TIGHT_MEMORY_CONTAINMENT allocation scheme.

Socket communication is now handled through the STREAM class which was extended to also wrap sockets. The low-level file and socket IO goes through the new HOST_OS::FileAccess, FileSeek, FileOpen functions. We're doing overlapped IO under Windows. Changed the meanning of the return value from Read/WriteBuf to be the actual bytes accessed rather than the bytes to go. STREAM constructs a socket when mode includes the 0x80000000 or 0x40000000 flag. With 0x400000000 flag the socket connects, with 0x8... the socket binds. Nesting a socket STREAM means to listen and accept a connection on that socket.

CAT_ABSTRACT initializes with a server IP and USER instances initialize with a default client IP. Both use the same port 2005.

STREAM::Text must be boolean to participate in predicates.

Resolved a problem with expanding the TypeInfo tree.

Started the CallReason logic.

Gaius5 constructs new types with the CLSID_TYPE identity.

Canvas streams had to override the ReadBuf method that works on files and sockets, and do nothing for device contexts.


161 Friday, March 04, 2005

The Morphic Core now loads the MorphTypeLib.TED editor collection upon init.

BASE {

Changed the unused STREE_BASE::TotalCount to Age. This new field represents the age of a tree node, relative to its parent. The age of a tree ticks up upon each Insert.

Added the new Edit virtual method that should invoke the Type Library editors.

Added the new HOST_OS_RT:: SetTimeReference function that sets a time reference from an internal absolute timer and returns time differences between the argument and the internal value. The resolution is milliseconds, and we're now using it for profiling.

}

CORE {

Added another test channel to load a type editor for the current node.

Finally understood why I couldn't increase the data space of the Morphic Core to give it more than 50M of heap space. The story is this: OS.DLL isn't linked into any EXE and gets loaded manually from the tray. Upon load, the system tries to find a memory hole for this module and fails. This is because of the multitude of DLLs that K5 Enterprise Navigator is using. The lower memory range that could have been sufficient (below 1G) was broken by INTRFACE.DLL at 0x08000000 or so. Had to move the interface much higher up, at 0x18000000. OS.DLL may now have a 512M BSS.

CollectInfo moved to TREE.

The new dictionary creation logic creates enterprise xrefs of hundreds of thousands of objects. The process may blow out of memory. Had to add protections against null pointers now arising.

Found a big piece of trouble in the memory manager in TYPE::CreateInstance. This should indeed use the nearest allocation scope for MemAlloc, but we're lacking still TIGHT memory, so that failed to link the heap chains resulting in leakage.

Added the DICTIONARY type master. Encapsulated all the dictionary free floating logic into this new class. Corrections, enhancements. Added strong profiling information and trace messages. The component responds to Default action with traces.

Needed memory allocation profiling for those dictionaries. Enhanced ABSTRACT::SelfTest with code 3, where profiling info is logged. Added the ABSTRACT profile counts, intercepted the memory management functions to increment the counts. Because ABSTRACT::DefaultAction is unavailable as taken by the LOGIN logic, we had to implement the DefaultAction of the MORPH category to route to ABSTRACT and run the memory profile test.

Found and fixed some problems in mmm_free, etc which didn't know that the memory manager pointer is ABSTRACT now instead of MORPH. Found a nasty problem in construction/destruction of volatile CAT_MORPH instances (this is something that the editors do when taking a proxy to the object) resulting in a crash due to the Autoload stream not initialized in SetTypeMaster. Fixed.

}

GUI {

Components: Destroying the window handle in the tree viewer was actually reconstructing it before destroying it. Found a case of mismatched style of memory allocation in the RTF viewer, in the OLE callbacks where memory was malloced and then GlobalFreed. Added the Cancelled flag to the CGauge control. Disabled the exception throwing.

GAIUS: Added the cancellation logic on double-click on the progress indicators. Checking this condition inside the loops and recursions of the compiler/generator. Improved the gauging in the generator based on items gone vs. items still to go. Code generator is now threaded. Got trouble with threads and RichEdit control (OLE). No updates to the RichEdit are possible until the form is visible so that its windows are constructed. Had to prevent object reloads until the handle of the current info pane is allocated.

SchematicEditor: Had to have XRef indications on nodes as well as types. Moved GetXRefDegree into the node editor. Canvases should not use the information field, because they should not malloc.

Changed the GUI affinity of the local editors to the upper-right panel.

Added the editor dispatcher to EditObject. Known editors: Hierarchy, Dictionary, Abstract.

Fixed the Apply and Undo endless recursion fault.

Added charts to ABSTRACT and DICTIONARY editors.

}


162 Sunday, March 06, 2005

CORE: Added another test channel to load a type editor for the current node.

GUI {

Launching the local editor is a TNullRelation::Reload on level 2.

Greatly improved editors for Dictionary and Allocator. In-place editors. Added charts and string grids.

}


163 Thursday, March 10, 2005

CORE {

Found and fixed a problem with collecting (serializing) information from a tree.

Started to write the HTML compiler based on a predefined set of TAGs.

Dramatically sped-up the compilation effort using a recursive limiting interval algorithm.

Added Options and Flags to the DICTIONARY type.

A dictionary flag turns off the encoder. Another flag turns off the preservation of original information.

A dictionary global option flag turns off the preservation of the original HTML/XML scope.

}

GUI {

The delete action of the Scope View is now multi-threaded. Deletion can be lengthy and we want to see the memory manager editor updating.

Added the generic flag/option field editor in the Object editor.

Improved the editor for the abstract category with a working grid, configurable sampling rate, and nicer default buttons.

Added the Options and Flags editor sheets for Dictionary.

}


164 Saturday, March 12, 2005

CORE {

Found and fixed a memory leak in STREAM::INIT with caching where the original file stream was destructed but not freed.

Added the new MemScan function to TYPE. This method actuates a callback. Primarily used in the new memory map viewer.

Added profile information for MemReclaim. This is currently the hardest effort of the Morphic Memory Manager.

TYPE::MemScan is much like SelfTestMemory but corrected in some aspectes regarding the iteration of disowned instances.

A DICTIONARY is now a specialized TYPE, preparing to become a memory manager.

}

GUI {

Implemented the Memory Map Viewer. It features a graphic area where each pixel is colored specifically depending on what blocks of memory lye behind. The map is built as a result of the new TYPE::MemScan callbacks.

Double-click on the map does a reverse correlation on blocks and dumps the content to the HexDump view. The map allows zooming in and zoom to selection, scrolling up and down in zoom mode. etc. The user may change the pixel size.

}


165 Monday, March 14, 2005

CORE {

Working on correcting the MemoryScan method. Discovered a situation where the callback wasn’t triggered.

MORPH::USER_GROUP wasn't declared as descending from TYPE resulting in a hole in the memory map.

CAT_QUEUE, CAT_POSIX, CAT_VIEWS, had to move to the end of the memory map for MORPH because they are un-initialized Also, pointers such as Debug are impossible to hop at this stage in a memory scan.

}

GUI {

Corrections and enhancements to the Memory Map Viewer.

Found an fixed a really treacherous problem where the height of the bitmap was wrongly calculated to be its width. The map was all screwed as a result.

We’re now distinguishing between header and data regions of a block, representing the header in light gray.

Clipping the beginning of a block that’s outside the visible area.

Showing the type of the current object in the status bar along with the object’s name. For a used or free current block we’re now showing its size.

We are more careful with the extent of a block in drawing operations.

Corrections to the workings of the scrollbar which has a lower resolution than necessary. Hinting the settings.

Some fixes to the selection mechanism.

}


166 Tuesday, March 15, 2005

CORE {

The dictionary introduces a new style of object construction, posing the following problem: can a type master create instances of itself during its creation? This is a recursive process not entered until now. OBJECT::SetTypeMaster was saving and restoring the object handle around calling the descendent constructor. Object handles (for STREAMS, NODE constants, THREADs, etc) might be preset in the INIT methods and expected to transpire the SetTypeMaster call. DICTIONARY claims a large free block in the SetTypeMaster method, and wants that handle preserved. I had to disable the handle restoration logic in support of dictionaries. Old logic may suffer. This is a subtle matter.

Found a major problem in the Morphic Memory Manager, which is actually a common problem of all heap managers: fragmentation due to large chunks. The story is this: a process loops to allocate a large block, then parse that block to create tokens, then frees the large block and keeps the tokens, and the again. The MaxFreeBlock cache pointer of the memory manager evolves towards higher addresses, where it conveniently finds the largest block, which is the heap top. When this runs out, the manager starts being more careful and looks for blocks from the beginning. The memory is already fragmented and the largest free blocks are those wholes resulting from the deletion of the large chunks. If a new chunk slightly larger than the others is requested, the allocation would fail. The matter gets progressively worse, following the same model when the large chunks are also fragmented. Avoiding this situation is really the responsibility of the client software, to order its steps more carefully so that the large chunk is the topmost block and deleting it would keep the heap top contiguous. We added another method to the Memory Manager that resets the cache, forcing the free block lookup from the beginning of the heap. The new MemResetCache is made available to C programs along with malloc as the new memreset C function. One would need to call this function right after the de-allocation of any important chunks of memory so as to temper the manager's impulse to use the largest free block at hand, or after the mallocation of a large chunk so that the subsequent smaller allocations would thoroughly go through the heap from the start. An important place where large chunks are claimed and freed is around the STREAM cache.

We're now saying that the destructor of a Type Master is also in charge of "destroying" the amorphous blocks. The term destruction is improper, as it applies only to objects while free applies to amorphous blocks. This issue was first posed by dictionaries.

Block adjacency in the Morphic Memory Manager is a critical issue. Inter-block gaps that aren't mapped by type masters can be disastrous. Such a black hole can form with a misbehaved type master, or a type master that isn't fully self-aware, and reports a wrong instance size. The problem is important as it leads to un-recoverable fragmentation. Had to add log messages to alert this occurrence.

Found and fixed another major flaw in the Morphic Memory Manager. The object Reclaim method was failing to traverse the whole necessary length of the allocation chain in search of an object's neighboring used or free blocks. This is a flaw introduced by the caching scheme after which the assumption that ALLOCATOR::Children points to the last free block is false. This resulted in tremendous fragmentation and even corruption of the chains leading to strokes.

MemScan is now spinlock protected against contention from the other MMM functions. Hard thinking has made me want to strongly order the scan left to right. This is to favour the coherence of the Memory Map Viewer, which would be upset if small order blocks would start calling back to draw after high-order ones. There is the penalty of first having to unwind the cache to the beginning of the heap, with potential for trouble for a certain type master that depends on a special order of its instances. Such an allocator either isn't an allocator, or it should not MemScan. MemScan now returns the size of the largest free block in the heap.

Added another profile counter to DICTIONARY to measure the total XML text attempted for wrapping, that collects even when the dictionary is disabled. This is for simulation.

}

Memory Map Viewer {

Added the ZoomOut feature. Added the system of book marks. A combo box gets filled with distinct memory addresses upon any double-click on the map view. One can then pick an address from the list an refocus the map viewer to it. Double click on the bookmark combo clears the list. Added a configurable memory refresh timer. Added a flag to have the Map Viewer auto-track the last memory chain modification, as indicated by the allocator's free-block handle. This is very useful in monitoring a memory process. Added new logic to handle the large steps of the viewer's scroll bar. Enhancements to the drawing of the map: we're now showing the headers of free blocks in white-pink and the header of used blocks in white-green. It was important to distinguish between header and data so as to know where to start the memory dump (dbl-click). Corrections to the drawing of the map: a more sane clipping logic. We're not filling the bitmap in black, saving some time and avoiding some flicker. Selection seems to work now. We're now using multiples of PixSize for the width and height of the bitmap, while stretching it to fit the control. This eases some calculations. Zooming in now stops when 1:1 bytes/pixels is reached.

}


167 Friday, March 18, 2005

CORE {

Every now and then, after extreme effort, the development of ANNA K5 falls into a neat place where everything comes together. This was one such moment with the following highlights:

  1. Generalized configuration is now possible and used at certain key places in the ABSTRACT hierarchy. This was one of the most important aims of the K5 development.
  2. The first functional morphism was induced when the Morphic Type Masters Editor family installs and overloads the Core's virtual method tables. This is one of the aims of the K5 manifesto.
  3. Generalized profiling has now been accomplished at the Core level.
  4. Structured memory allocation has now become possible and is currently demonstrated by the DICTIONARY class which is a sub-allocator. One of the major aims of the ANNA K4 development was "frontier negotiation" between adjacent memory managers. Although still under development, this revision of the Morphic Memory Manager exhibits certain aspects of such boundary negotiation.

In detail now:

  1. Every type master declares a SProfile sub-structure that follows the same descent vector. A type inherits all the profiles of its ancestor. The matter of lateral containment is unsolved yet.
  2. A new OBJECT_BASE virtual method called GetProfileInfo(names, length) has two uses: one is to return a profile structure, if the object instance has one; the other is to return the names of the profile vectors. You call it first with a null names buffer in order to retrieve the buffer length. You then call it with an allocated buffer of the required size to get the names which are new-line separated. You then call it repeatedly with null names to get the updated profile data and the count of profiles in the length pointer. Each type that introduces profile counts must override this method to return their names. Certain descendent types also allocate profile space and would also override this method to return the pointer. CAT_ABSTRACT and DICTIONARY are such classes that allocate profile space for their instances.
  3. One effect of the new profiling scheme is that ALLOCATOR doesn't need to overload the MemAlloc, MemFree, ..., MemScan methods to update the profile counts, and TYPE can do that even though this type doesn't allocate profile space of its own.
  4. DICTIONARY is now modeling ParseHtmlSection. The dictionary is now a memory sub-allocator creating one string for each new word. This is just for demo. Fixed a crash condition when canceling the multi-load process.
  5. CollectInfo is now called with a flag that drives the inclusion of new-lines into the collected text.
  6. OBJECT::Edit is a static virtual method.
  7. Dictionary overloads GetStartAddress to return the managed block.
  8. There's a very new and powerful little function TYPE::InstallMethod. This is the start of the functional morphism. The method is called to overload a type's method. It iterates the DNA looking for descendents of the given type, and overloads them as well, if they haven't already been changed. This method is used by our new scheme of generalised configuration.
  9. I'm still vacillating over the matter of ALLOCATOR freeing its amorphous blocks. (Reverted for now)
  10. Found a case where the MemScan wasn't calling back for a certain block.
  11. Found a major problem in the MemResetCache function that was looping endlessly for type masters that only had object instances.
  12. Protected GetStartAddress against a crash condition when the container is null.
  13. Found a problem in the layout of the OBJECT_BASE VMT at the point of set_Name, which must always be last. We couldn't register the editor type library because of that, nor could we call each editor.
  14. Cught a lousy problem in the Compiler category in that TExpression wasn't binding to its RELATION ancestor.
  15. Added a nice epsilon face to the editor buttons and menus.
  16. The command console was extended with a GuiPeekDispatch loop while not kbhit. This has made possible the launch of the new Morphic Type Library Editors from the command line.

}

Morphic Type Library Editors {

We don't need to override the core's CreateEditor method. Anyone can now call the local editor of an object, without specifying a docking container frame. The caller must, of course, allow GUI messages to be processed. This is a major quality difference between ANNA K5 and VCL in as far as object configuration is concerned.

Generalized configuration is one of the most significant steps forward lately. The type editor is a recursive structure of tabs contained in an abstract frame. The frame is EditFrame and has the following elements:

  1. Bitmap 48x48 drawn by the type master of the currently loaded object
  2. Apply, Undo and Help buttons that drive into the chain of recursive tabs which are specific to the type of object.
  3. A progress indicator that anyone can use in whichever way, having the nice property that it would always be visible regardless of how small the viewing area is for the editor.
  4. Finally, a page control where each speciffic type editor will recursively insert what we call aspect editors.

Each aspect editor descends from the abstract TForm_AspectEditor.

A type editor for a type T descends from the type editor of its ancestor type A (where T->A). The constructors of these frames progressively add aspects to the page control.

The following editors are currently defined (because I need them): OBJECT, TYPE, ABSTRACT and DICTIONARY.

The editor for OBJECT introduces an aspect viewer for profile information. The new profile viewer (Grid for selection and Chart for display) formally belonging to ABSTRACT, relies upon the profiling system now generalized across the Morphic Core with the GetProfileInfo OBJECT_BASE method that returns the names and values of these profile vectors. Nicer GUI for this editor. Logarithmic chart display and on/off for legend.

The most important aspect of this new Morphic Core is the InstallMethod fetaure which is the beginning of Functional Morphism. The editor library calls it to install type editors at the level of the virtual OBJECT_BASE::Edit method. Object types that don't have their specific editor would still inherit those for their base types.

The editor for TYPE introduces as its specific aspect the Memory Map Viewer. The memory map is only visible if the type master's memory map is non-trivial

The editor for dictionary introduces a new aspect for editing dictionary flags and options. It inherits the memory map viewer from TYPE as well as the Profile Viewer from OBJECT.

}


168 Sunday, March 20, 2005

CORE {

But the grief in between... I have problems in the Morphic Memory Manager as exposed by the DICTIONARY.

Had to implement the MemRealloc method of TYPE so as to lower the fragmentation inside the dictionary. The MemRealloc is the union of MemFree and MemAlloc with a possible buffer copy in between. MemAlloc and MemFree are entry points into the reallocator. The problem now is that object instances must have a determined size so that they may be hopped-over by the scanner and integrity tester. An ordinary used block may be larger than the required size is no right-hand-side free block can be formed. Object instances need to be flagged as either being contained in other structures, and therefore not including an internal fragment tail, or as not being contained, free-floating within a memory pool, and therefore rounded-up to the largest AMORPHOUS header size multiplier. A free floating object instance can be allocated only if an exact hit is found, or if a right-hand-side free block can be created. These conditions are hard to match with high fragmentation. Still experimenting and thinking. For now, we had to properly mark object instances with the OBJECT_BASE::flNotContained flag.

The DICTIONARY has changed with FindWord, Encode, and Decode taking and returning abstract handles to words, which may either be NODES oar SWords depending on the dictionary operation mode which is configured with the flB2 flag. The old dictionary mode constructs words as NODEs and grammar links as XREFs. The new mode uses the SWord structure allocated off the local dictionary heap space. SWord is a variable size structure consisting of a header with word label and size, followed by a variable length list of links to other words. The dictionary uses indirect pointers through the WordTable, so the words may be relocated without the penalty of re-basing the whole memory map of the dictionary.

}


169 Tuesday, March 22, 2005

CORE {

Improvements to DICTIONARY. The dictionary is larger than 64K words. Had to change the encoding scheme so that the links have a configurable size. The dictionary now has the Sizes static options where [0] is the maximum number of words, and [1] is the number of bits required to represent the links of such a dictionary. Current capacity is 1M words. Tried some different ways to improve the speed of the encoder. One is to order the blanks so that those more often encountered are closer to the start of the blanks string. Tried to avoid calling atoi senselessly. FindWord isn't getting the token's length, just does a strcmp expecting the null terminated name. Word labels are now null terminated. Corrected the Sort function of the dictionary. Slightly sped-up the encoder and decoder by pre-calculating the size of the Links index of each word, rather than shifting bits every time. Some significant speed was squeezed by changing the orientation of comparisons in GetToken. Each input char is tested against the blanks. This avoids strlen on the entire input band, and potentially long lookups for blanks that are further away than others.

A lot more work on the Memory Manager, which has now become a really elegant and transparent piece capable of de-fragmentation. After intense pain tracing all those memory corruptions, I've found this silly mistake, not even a student would make, not wrapping the body of my_min macro in parenthesis. This is why memory relocation with memmove was overrunning by one header size.

MemRealloc is now a much more powerful function that can (and is) used for de-fragmentation. Added another new_ptr argument to it to tell the Relocator that the target free area of the relocation is already known. The de-fragmentor MemGrow shifts used blocks over the previous free blocks using this technique. MemRealloc eventually starves, then calls MemGrow to find a free block one way or another. MemGrow may do a de-fragmentation pass over the pool until the required block size is satisfied. However, Realloc may have not been called from MemAlloc but directly, to re-size a memory bock. In this case, de-fragmentation must not violate the original block content. To achieve this transparency, we made it so that the largest free block of the heap is not necessarily at the end of the memory map, and may actually rotate around the chain. MemRealloc in conjunction with MemGrow makes sure that the block to be relocated is wrapped in this pivot free block. The pivot free block is an allocator's last instance.

Realized the MMM_SPINLOCK was global and therefore shared between the ABSTRACT and DICTIONARY allocators with confusing results. Introduced the new TYPE ::flInMemRealloc flag used by the new spinlock. Organized the logic that contends over the memory pool to use carefully this flag for mutual exclusion. Re-entry is permitted only between MemRealloc and MemGrow. DICTIONARY::MemGrow is the new dictionary defragmentor. DICTIONARY didn't need to override any of the other memory methods. When the dictionary encoder grows a word's link list, the entire dictionary may be re-based as a result, so none of the pointers can be trusted. Had to reload sword as well as slast_word. Dictionary doesn't need to reset the cache because it can de-fragment its heap. .

}

GUI: Fixed a problem in the multi-relation when launching the schematic editor while a process (Scope Browser changes) tries to refresh a schematic which is still null.


170 Thursday, March 24, 2005

CORE {

Enhanced TREE::CollectInfo with the b2 flag that instructs it to add parent info to the end after the children. Our HTML parser constructs trees with this kind of information affinity.

TYPE::CreateInstance had to leave the handle field unchanged when no data size was required. This is to support the dictionary which sets the handle (claims a block from the system heap in order to sub-class it). The serializor of dictionary calls that of TYPE which must ignore amorphous blocks. Introduced a test.

Found ill-formed HTML files that have end tags for no corresponding start tags. Implemented a counter scheme for each tag type to check for the condition and ignore it as if the end tag was a series of spaces. A new function to remove form-feeds and page numbers from input text.

To serialize the dictionary we had to make words fully relocatable. Name had to turn to NameOffset into the LabelsString.

Added a new function to log the new words of a dictionary as of a certain index.

No more special words defined in the constructor. The null is not a word and rather a reserved WordTable position (0). Lambda is FindCreated in Set/GetInfo, so is <number>. This too is in support of the serializor. The dictionary serializor streams the sizes, then the labels vector, then each variable-length word structure. The Hierarchy aspect of a dictionary is serialized after fudging that the allocator has no blocks. Applied a new heuristic to FindCreate and searching the links of the previous word backwards. This is based on the assumption that a language context is generally coherent, and therefore more likely to take-up again terms recently introduced in the near vicinity. Corrections to avoid WordCount[0] which is 0. We're substituting the XML characters now.

We automatically sort and store the dictionary every 10 files wrapped.

}


171 Saturday, March 26, 2005

CORE {

I've made another version of the compressor to correct the problem the previous one has with mismatched casing. The new one adds lowercase words to the dictionary. It uses an encoding scheme whereby the separators between words are excluded from the semantics of the language and coded apart between two word indexes. This new scheme forces at least one bit (we call it the "normality" bit) between any consecutive word indexes, so trivial grammar transitions are not allowed. Abnormality is anything that isn't a blank between words, or a different casing in the following word. In the abnormal case, more fields are encoded. The exception bit is followed by a 3bit exception type descriptor, and followed by a chain of 6 bit separator descriptors. Repeated characters are coded with a bit set in the MRT descriptor followed by a count. The type field of the MRT describes one of the most frequently occurring separators such as ", ", CRLF, and a few others. The more unusual separators are coded with an ASCII code following the MRT. I was very annoyed to find that this scheme produces lower compression rates (20% less), and doesn't reduce the rate of link creation as I had initially speculated. One advantage might be in the re-encoding aspect of the future algorithm where the separators are guaranteed to not be shifting. I can't tell yet how important that may be. Oh, and one more thing. The decoder for the new encoder doesn't exist.

Found and fixed the HTML parser who didn't expect the <H6> or any such heading tags.

Greatly improved the dictionary creator and encoder, halved the number of FindCreate calls by presetting the separator words and keeping them unmovable. Found a problem with GetToken that should return a 0-blank when the input band has run-out. Some text was wrongly encoding because of missing the EOF.

}


172 Sunday, April 03, 2005

CORE {

Another difficult engineering pass over the DICTIONARY. Enhanced the dictionary with the new mechanism of morphing the word index size on the fly, while new words are added. This is achieved with the new ResizeWordCount method. Developed the NEW_COMPRESSOR's de-compressor logic. Written a differential to test the identity of the encode-decode process. Wrote yet another encoder/decoder scheme defined by NEW_COMPRESSOR_EXCEPTION that isn't emitting the true normality bit and rather encodes an <exception> link in the stream. This was supposed to save the one bit between normally flowing words. However, the exception poses the following major problems: 1. eventually the exception links to and from every possible word in the dictionary and becomes its heaviest used key-word. This in turn makes for an exception index the size that tends towards the total word count size. With an exception indicator of nearly 16 bits (for example) the whole benefit of saving the normality bits is written-off. There is something worse yet. The grammar of the language was better-off abstracted from the details of casing and separators. We've already explained this in the previous revision, and this exception based scheme is against that breaking the language coherence. We eventually want to have the possibility of searching for a sentence ignoring separators.Added GlobalCompression profile field. Allocating the InfoTable and the EncodingString in preparation for the future morphisms over the encoding.

It's been over a year now since the development of constants and selectors, and shamefully must admit that I've forgotten how they work. Found a problem with constants in the Rose model loader. The loader creates constants based on documentation fields that indicate the size of that type. Constants appear to have the value as their handle, which is only valid if MaxBits is also set. Being a constant also meant that the node header couldn't have been bound to a container. That concept was limiting right from the outset, because we tend to think of the entire kernel with all its compounds as one big constant now. So constants are not necessarily free-floating headers, and they can also be contained in other objects. With the introduction of the structured Memory Manager, the node containment indications have now become unreliable in as far as segregating constants apart from selectors and general nodes. So there's now a problem serializing such nodes produced by our model loader. Fixed it in NODE::IsConstant by considering now that a constant is solely determined by a non-zero MaxSize field.

Found another problem with constants generated by our model loader stemming from ABSTRACT's "new" operator. The operator was deferring allocation to the original un-intercepted calloc function, and thus not on our heap and not capable of realloc as required by the NODE serializor.

Repaired the Rose model loader after that change to the separators of TREE::FindCreate.

}


173 Friday, April 08, 2005

CORE {

Found a problem in the ABSTRACT::bit_memcpy function which wasn't prepared for copying overlapped regions. This is analogous to the problem of memmove only worse because we have to do bit manipulations within each byte. This algorithm is slow and absolutely MUST be sped-up.

}


174 Tuesday, April 12, 2005

CORE {

Bit string moving is the key to morphism and must be fast. A processor should provide accelerated support for it. The new bit-string mover has two versions, a reference one written in C and another in Intel assembler for speed. The algorithm is subtle as the algebra of the numeric domain is quite complex. The algorithm must ensure associativity of bit concatenation. bmemcpy belongs to the K5 ANSI library, but is our own extension to the standard. A Core built with a foreign ANSI must link in bitmemcpy.c explicitly.

Algorithms such as this should be started with a serious test suite that identifies all the important scenarios of the algorithm. The test suite should be devised so that all the boundaries of the algorithm will be crossed. In the absence of such a test suite it took me 5 full days to tune this algorithm, and finally the result was possible only after a comprehensive TestBitMemCpy was implemented. The assembler version moves a whole byte at a time. There's potential for a routine capable of moving 32 bits in each iteration using a complex carry register, but that's for later. Most computer processing is byte-granular. Our dictionaries and Enterprises by contrast are bit granular. There wasn't much scope for bit-stream shifters until now.

Dictionary has evolved to store the encoded strings. This is one of the most sophisticated matters in the entire system because it is an aggressive application of structural morphism. This is how it works. A dictionary has two major areas that it must manage: the grammar (words+links) and the encoding. The grammar suffers structural changes as new words are added and as the grammar grows with new links. When a grammar production gains more weight, the index size of that production changes. Any bit-stream encoding this index must be re-encoded at the places of occurrence. In other words, the encoding must morph. The encoding stream is an area foreign to the dictionary, owned by outsiders. The grammar is the DESIGN whereas the encoding is the DATA of the dictionary morphic program. At a certain point in the compilation process, a new grammar link is added between two adjacent words W->Wn. W however has occurred several times before in previous texts. The entire encoding must be re-evaluated, all occurrences of W must be found and their bit-stream offsets collected, then at all such places the stream must be shifted right by a compounding amount to make room for the new index of W that is larger in size. A tremendous amount of shifting is required as W is a more common word. The new dictionary does all this. There is only one encoding pass now, which does everything from adding grammar to returning the required encoding length and the actual encoding as well. Conversely, there's just one decoding pass now that either decodes an encoding between two bit-stream positions or compares the would be decoded text against the input text. The new decoder is therefore also the differential and, as such, triggers the difference report. The GetInfo routine is also the word finder, and returns a vector of bit-stream locations occurrences of the given "resizing_word" argument in the encoding. This word-finding mode is used by the ReEncode function to find all the places onto the output-band that need shifting. The dictionary morphing algorithm is extremely slow. Turning the bitmemcpy to assembler has sped-up the process, but this isn't enough. All objects need room in which to grow. This isn't as much for themselves, as is for all the neighbors that want to be left alone and not be pushed around by others. ANNA K5 is in essence a 3D memory manager.

Explicit dictionary options and flags: ofKeepHtmlScope, flDisabled, flPreserveSourceInfo, flEnterpriseDictionary, flVerifyEncoding, flVerboseReEncoding. New dictionary methods: ReEncode, DeEncode. SetInfo and GetInfo take a reference to the bit offset. They use and update this value. New flags field for the unified DeEncode dictionary method. ESetGetInfo defines the following reasons for calling DeEncode: sgSet, sgGet, sgVerify, sgReEncode. FindWord has dropped the new_sentence argument.

New profiles: ReEncodings, ReEncodingGrowth, ReEncodingBitShifts. Obsolete Min/Max Compression/Expansion profiles.

Added the new bmemcpy function to the ANSI library. OBJECT::bit_memcpy calls it in fast mode.

}


175 Wednesday, April 13, 2005

CORE {

Sped-up the dictionary decoder by preconfiguring the WordTable with lambda, normal numbers, dot-separated numbers, comma separated numbers, and Punkt. Once created, these words may be found at their absolute locations without searching. This avoids an expensive search at the top of GetInfo and a 32*3 loop to discriminate each word against numbers during decoding. The sorting routine is now broken. Speed is better for re-encoding but we need something much more serious here.

Sped-up the ReEncoder by a factor of 2 (loosely speaking) through a series of moves such as duplicating the GetInfo logic inside ReEncode, stripping any text-oriented logic out of ReEncode down to the minimum necessary code to skip over separators and build the relocation table, and going directly to the bitmemcpy function avoiding another stack level of OBJECT::bit_memcpy.

Gained another 5X speed increase factor with the following strategy: information is now treated as a word and added to the word table. What discriminates words from infos is that words have a non-zero NameLen. Infos store bit-stream positions as their NameOffset. The new SetInfo creates an info node with the beginning of the encoding stream for this info and propagates it through Encode all the way to ReEncode. Here, the following assumption is applied: any words added to the word table after the context info node must have been recently introduced during the analysis of the current info, and therefore they cannot have occurred in any previous information encodings. It is therefore not necessary to re-scan the entire encoding from the beginning looking for posterior words. This assumption has greatly diminished the number of re-encodings (morphisms) yielding a speed close to the previous version of the algorithm that wasn't persisting the encodings. We must go much further now and determine exactly the limits of these re-encodings by keeping the bit-stream offset of each word.

Another principle: a word is an encoding of letters stored on the encoding stream at the place of its first occurrence.

}


176 Saturday, April 16, 2005

CORE {

Coded the REM MOVSD in bitmemcpy for those lucky cases when source and destination are byte aligned.

Information strings are coiled onto the dictionary as words. No need for an InfoTable anymore. Passing the information node through the Encode/Decode functions.

Diminished the reliance of the dictionary algorithms on the SWord::Index field. The only one user of it is the MemGrow algorithm that needs to update the WordTable after relocation.

Hidden away the NameOffset and NameLen details of SWord in preparation for the next revision of the encoder which would use lexical finite automata to match and form word labels without storing the ASCII strings. The encoder/decoder use the new GetName and NameCompare functions. The dictionary initializer also emits primitive encodings onto the EncodingString. Info encoding can never start right from the beginning of the output band. Added logic to the initializer that could create the alphabet. Obsoleted the numbers vector in the Get/SetInfo. Changed the encoding of the mixed-case exception so that its encoding precedes the link index. This was so that any party pointing to a link index on the encoding may start de-convolving forward. The mixed-case exception required knowledge of the previous EType field, and could not be parsed.

A major change and improvement in the dictionary. Words have a new BitPos field that points to the first place of instantiation in the encoding stream. The data morphism algorithm ReEncode needs not process the encoding leftwards of this value for any resizing word. Early words are either common ones that would suffer rapid changes, or rare ones that would evolve slowly but effect longer runs of re-encoding. We must observe that if a heavily used word is old, then it would slow down its rate of growth as it accumulates connections, and re-encode less often than the younger words. The young ones, as in life, are most rapidly changing, but their effect on the encoding is limited towards the tail of the encoding. This refinement minimizes the length of bit-shiting. The ReEncode morpher function has now got a new responsibility, which is to re-base all the BitPos pointers after a data morphism has occurred. The WordTable is now kept strongly ordered in the direction of BitPos so that ReEncode would not need to re-base any word below the resizing one. A word morphism therefore affects only the following words in the WordTable order. This is a refinement that affects the effort of pointer re-basing. The whole DeEncode-ReEncode process must be clockwork, and changes here are dangerous.

There's another way to diminish the length of morphism (and not its rate) by providing sentences with some internal fragmentation. Each individual needs a bit of space around to grow in. This precept isn't for the sake of the individual that needs to grow, but for the sake of everyone else around him that mustn't be disturbed. This is an important principle of the Morphism Core.

There's a functional correspondence between the DICTIONARY and the ABSTRACT core made of OBJECT and TYPE.

Gained another 20% or so performance in the dictionaries by coding in assembler the bitmemcpy exceptions of alignment for byte and word, not just dword. This seems to have helped the multitude of int copies which are more often aligned. Coded in assembler the GetIndexSize function using a 386 BSR instruction. I couldn't observe much speed improvement here.

SWord is somewhere between AMORPHOUS and OBJECT. It needed a GetTypeMaster function to return the containing dictionary. We avoid senselessly passing the dictionary pointer through the SWord methods. This research into dictionaries is getting more interesting. A dictionary seems to be another kind of Morphic Core. This is only normal, since a dictionary is a file system, and that, in turn, is another instance of ABSTRACT persistence.

Moved the responsibility of creating info nodes over to SetInfo. This move is in preparation for recursive dictionaries and alphabets, which is coming-up next. SetInfo now returns the word index for the info node created. Changed the meaning of bits and reference_word within GetInfo, so to mirror SetInfo.

Increased the compression rate by some 10% allowing trivial grammar transitions again. This was thought as impossible with the new exception scheme, but tested just fine. It is also needed by the future alphabet that needs lambda productions.

Cleaner LoadXML method, abstracted from the inner types of the dictionary.

}


177 Tuesday, April 19, 2005

CORE {

Bit-shifts are expensive. The new GeRELATIONSize function wraps the GetIndexSize and bounces the value against a look-up table of larger numbers that will determine some internal fragmentation in the word encoding, thus avoiding morphisms for a while. The worse case scenario is encoding word links as 32-bit values. This yields a compression factor of 1.4.

Started some more bit-shifting logic for those cases where the sofs and dofs are aligned, and could use a MOVSD. The story here is more complex than we now have time for.

Compile options for PUNKT and for NUMBERS. Lambda is null. Modeled the SWord class.

occurrence

}


178 Friday, April 22, 2005

CORE {

This research is getting harder.

Added another flAlphabet flag to the dictionary to set the de-encoder to a new mode of letter munching.

This version of the dictionary has several modes of operations. It works creating NODE words as well as SWords, so the original operation mode has been repaired. It has a new alphabet mode where words are letters and therefore tied in a finite set. The maximum speed of the encoder is reached in alphabet mode, where no FindWord is needed. This was the major un-balance between the encoder and the decoder with the former searching encoder taking much longer than the decoder to parse the text. The dictionary constants now reserve the first 256 positions for the ASCII codes of the input text so that the lookup is instant.

Alphabet mode is a recursion of the word mode and would normally require a dedicated dictionary. Having two growing dictionaries in the system would be prohibitive in terms of memory reservations, so the new dictionary is recursively both and uses one heap only. A more difficult start-up grammar had to be tuned in the Initialize method so that both alphabet and words may co-exist in the same dictionary. The new dictionary has <null> for letters and <lambda> for words. Lambda is the last symbol of this pre-defined grammar. SetInfo and GetInfo have special, tighter loops for alphabet mode, where there are no numbers, separators, punkt, etc. The new Encode method sets the first occurrence pointer for each word/letter. Added debug in Encode/Decode to show the grammar productions onto the log. Fixed the ReEncode method to work with nodal dictionaries also.

Found a major problem in the workings of the pointer re-base logic of ReEncode to do with aliased pointers. The story is like this: a whole string of 0-size encodings may overlap at the same bit position. Then, a morphism occurs in the middle of them. All those types (words) before the resizing one in the order of appearance must stay the same, all those there-after must shift to the right to make room. The difficulty is in un-tangling the overlap by the right amount when more than one segment (occurrence) is shifted. There's a very delicate "if (old_bitpos==old_pos&&prev_size) marker->BitPos+=dsz0; else marker->BitPos+=dsz1;" there that must be carefully understood. It took some 40 hours of tracing to finally tune the concept.

Another major piece of trouble in the ReEncoder now with this alphabet/word mixed mode is sensing the limits of the encoding between what words/letters are separated or concatenated. If the encoding pointer is below the Lambda position, then the words are concatenated letters with no separators, and separated with normality bits otherwise to the right-hand-side of Lambda otherwise. This test works in conjunction with the Initialize method where the constant grammar is being set. In fact, Lambda marks the end of the alphabet dictionary and the start of the word dictionary over the same encoding band, in a recursive dictionary.

Introduced another test that cuts-out as soon as all the word instances were found on the encoding, without decoding all the way to the limit. This may speed-up the re-encoder for old long forgotten words that now re-appear. Fighting for speed takes all these little moves like such.

Reverted the de-encoder initialization back to an older scheme of returning the Lambda, Punkt and Numbers vector. This helps isolate the differences between a NODE-based dictionary and an SWord based one. Added the SWord::CreateWord method to the dictionary, again to minimize NODE-SWord differences.

The Initialize method encodes a trivial path between all letters of the alphabet and all pre-defined symbols of the grammar so as to fix their first occurrences and therefore strongly order them with any new created words. Lambda is non-trivial and must have size.

}

GUI {

Added a valid_mask parameter to FlagFieldUpdate to avoid the recursive effect when setting an entire bit field that bounces beck to check-box clicks. Dictionary editor is now loading options and flags upon creation/Undo.

}


179 Monday, May 02, 2005

CORE {

Eight days of intense struggle to make this dictionary recursive. The new mode of operation is to treat text, words and letters all alike by recursing the SetInfo and GetInfo functions. Words are no longer stored in the LabelsString, they don't need the NameOffset and NameLen fields. Just as a text wraps around the dictionary as a string of encoded references to words, so do words encode as references to characters. 0-255 characters are the alphabet of the grammar. This 2-leveled dictionary would have normally been achieved using two dictionaries, a low-level one for words and a high-level one for text making references to the first. Since both dictionaries would have been subjected to intense morphisms, the word dictionary morphing due to inter-letter index size changes, the text dictionary due to inter-word index size changes, room for both memory managers would have had to be reserved yielding a large amount of wasted inner heap space. The solution was to get one single dictionary handle letters, words, and text in a unified manner over the same heap. The encoding scheme for words was the key to this. How does it work. A new piece of text coiling over the dictionary isn't known in advance to make all references to formerly introduced words, and may introduce new words to the dictionary. The new words are exceptions to the reference encoding scheme. The encoder must check to see if the word label is already in the dictionary and if so then reference it, or create a nested grammar for the new word otherwise. A general text encoding is therefore a sequence of references to existing words interlaced with nested grammars for new words. This nesting posed the following major problems:

1. The ReEncode morpher routine had to be taught how to navigate this new kind of interlaced encoding stream. The key to nesting grammars was to first encode a transition from the last word node to NULL, then encode the grammar of the new word as a series of productions starting from NULL, A, B, C, .. etc, followed by a transition from the last letter of the encoding word to the word node itself, then a transition from the word node to the last word node that was previously encoded, and finally a production from the last node to the new node. The rationale is complex: we need a way to introduce sub-grammars without marking each word separator with another exception bit. Encoding an exception bit to discriminate between reference and definition would adversely affect the compression factor all throughout the database. So we wanted a careful scheme of transitions in and out of the nested grammar that works for definitions just as it does for references, but allows the ReEncoder to sense when it should enter and leave the nested mode. Now, the first transition of last_word->NULL is required to indicate the definition exception. When the ReEncoder finds that a word transited into NULL, then it knows that whatever follows is a new word definition. The new word starts from NULL->first letter, so as to identify the first letter having NULL as starting point. As the letters are inter-linked the sub-grammar for the new word is both created in the word allocation system, and instantiated onto the encoding band. The last letter transits into the word node so that any decoder following a letter sequence may eventually find the word that has encoded it. Then, a more subtle matter is the transition from the word node back to NULL. We do this so as to create an instance of the new word right after its definition. This was needed for 2 reasons. First, if the new word wasn't instantiated here, then it would be instantiated soon when the next word would be encoded. This may happen after another sub-grammar was introduced, after separators, and other variable-length objects where encoded onto the encoding band. There would then be no reliable way to determine the actual length of an encoding for a word, other than by introducing another field to the word structure. These fields of SWord must be kept to a demonstrable minimum as they mean a lot more in the overall morphic operating system than the dictionary suggests. Every SWord field must be justified with respect to the entire system, and ELen isn't. The word instance right after its definition is a good way of marking the length of the sub-grammar for this word. SWord had to be extended with the Definition field which points to the first NULL->letter transition for the word. BitPos-Definition yields the length of a word encoding. Another reason (and related) is that ReEncode needs a criteria to sense the exit from a nested word definition. This criteria could be the return from letter mode back to NULL. However, we also need to restore the last_word pointer and encode the last_word->new_word transition by reference as usual. To re-gain the last-word pointer we must encode it into the definition of the new word. These matters are to do with the re-synchronization of the ReEncode routine when coming-out of the nest. We encode a transition from new_word to last_word here, which instantiates the new_word and also re-loads the last_word, and follow with a transition from last_word to new_word which is the normal requirement of the upper-level text encoder that works with word references. In short, the transition from last_word to NULL enters the nesting mode, the transition from new_word to last word exits, the transition from last_word to new_word is normal as if no nesting has taken place. SetInfo and GetInfo had to change accordingly.

2. The other problem posed by the new encoding scheme was the kernel constant grammar set in Initialize. A very careful set of transits had to be arranged here to define the NULL, the alphabet, and the Lambda symbol, in accordance with the expectations of the new ReEncode algorithm.

FindWord is now vastly different and doesn't match on word labels anymore since they don't exist, and rather it recurses into the decoder to verify the given token against the previous words. Decoding all the previous words would be prohibitive, so we have a new SWord field (which was grudgingly introduced and barely justified) called TextLen that tells us how long would be the word when decoded. This field is set by at the end of the Encoder positive for words and negative for text so as to also discriminate words and info nodes. FindWord uses this field to do trivial rejection of text nodes and of words that don't match that of the testing token. For all words of the same length the decoder must be run in verification mode in order to accept an existing word.

The resulting algorithm is much better logically and more efficient in terms of storage requirements, but unfortunately slower. We need another encoding scheme that uses the Encoder to also match duplicates.

}


180 Thursday, May 05, 2005

CORE {

Implemented the new and hopefully final encoding scheme whereby the encoding for a text contains the encodings of new words and references existing words. New words are introduced by a prev_word->NULL transition, the definition of the new word follows as the NULL->Letter1->Letter2->...->LetterN->new_word->NULL series of transitions. There is no need for encoding the last_word->new_word reference anymore. There were changes to ReEncode, Initialize, FindWord, SetInfo and GetInfo to support this scheme. The net result of this batch of changes is that SetInfo now assumes, when called recursively for a token, that the word won't be found in the dictionary. At the end, SetInfo does a search of all previously encoded words and if one is found to match the current encoding then its semantic side-effect is unwound. The new workings of SetInfo obsoletes the FindWord method. This algorithm finds words quicker, and has better potential for future improvement with comparing encodings rather than their text.

}


181 Friday, May 06, 2005

CORE {

Developed the bmemcmp routine in our ANSI. Uses REP CMPSD in the lucky case where of no alignment, or compares 24 bits at a time in a general scenario of longer misaligned data runs.

A whole heap of corrections and changes to gain speed actually leading to a slower system, I don't yet understand why.

Minimized the initialized variables of functions. BitPos is never -1 so that the test in a tight loop of Rebase isn't needed. No speed gained there. A tighter loop in SetInfo dedicated to letters built straight from the info character. No much gain there either. Obsoleted the FindWord method as SetInfo now recurses. Corrected the duplicate info finder in that encoding length need not be identical for two infos with identical output. This is due to the fact that the first one may have nested words contained and the second (the duplicate we want to remove) encodes these words by reference. The only way to see that they're the same is by comparing their outputs. Words that have equal length encodings are further compared bit-wise using the new bmemcmp function that we wrote in assembler. (hardly any speed gain there) Corrected everything to do with profile counts and internal Set/GetInfo counts for sane reports and for exact trivial reject of duplicates. Found a problem in GetInfo to do with reaching a limit but the encoding has one or more trivial transitions that won't exceed the limit and need to be walked through. GetName returns some bracketed name patterns for the constant grammar symbols. Such names must not be taken into account in GetInfo. GetInfo runs to the very end of a word sequence until NULL. Realized that ReEncode had to start from one letter before the resizing one to make it through any possible null transitions this was the source of intense grief and ambiguity. Now, whatever is below Lambda must be backtracked by one letter before entering the stepping loop. Tighter stepping loops to no observable speed increase. Changed the initializer to quickly bail out second time around and subsequently. The encoding of Lambda was wrong. Resolved the NUMBERS mode of operation, but strangely the compression factor is worse when using it. It is also true that this version of the dictionary starts off with fairly weak compression ratios as words are contained, and gradually improves the compression as words begin to be referenced.

Found and corrected the performance degradation problem. It was due to wrongly starting the re-encoding process from the beginning of the encoding string no matter where the resizing word had its definition start. Reduced the useless encoding function arguments. Cleaned-up old build macros because variants of the dictionary need a total re-think.

}


182 Saturday, May 07, 2005

CORE {

The struggle for speed. Found that a major performance penalty was SWord::GetLink and SetLink as the WordTable was bit aligned. This impacted on the algorithm looking for existing words. Changed Get/SetLink to be byte aligned, and to not go through bmemcpy routine because the link indexes are integers anyway and the discrimination logic of bmemcpy is worth only for longer bit runs. Never seen the encoder run so fast after these changes. Up to 10k/s. Added a checksum field to SWord that SetInfo sets and checks to trivially reject existing words. Found a problem in ReEncode that was preventing an encoding scheme where a 0-sized index encodes in one or more bits. On the left of Lambda, for letters, we must start from the BitPos and not from the Definition of the previous letter. Had to reinstate the -1 BitPos and the check in ReEncode's rebase loop, as we can't yet set the static grammar correctly in all configurable compression modes.

Added the Auto Save flag and logic so that the dictionary isn't necessarily saved. Added the ProfileInfo to Serialize.

Some more performance was gained by avoiding BITMEMCPY for short bit runs. Encode/Decode now use the new EncodeInt/DecodeInt functions which are faster than BITMEMCPY as they take advantage of the known size and alignment of one of the operands. The normality bit, etype field, mrt, and numbers are now also handled by more direct decode/encode functions. I think that we're reaching the speed limits of this concept.

A bug. EncodeInt can only be used to add to the encoding, not to modify it in the middle, because it is violating the end limit of the encoding. ReEncode must use BITMEMCPY to put back the index after a growth.

A little more speed by reordering the trivial rejection tests in SetInfo and tightening the loop. Some gain also by not calling ReEncode when the index size hasn't changed. Removed all breakpoints from tight loops.

Encode/DecodeInt are limited to 24 bits because of the shift offset. They cannot be used to Encode/DecodeNumber. Renamed these function with the 24 suffix to be explicit. Reverted to using BITMEMCPY inside Decode/EncodeNumber.

}


183 Tuesday, May 10, 2005

CORE {

Found a problem with these new Get/SetLink functions that were corrupting the memory allocation structures for a WordSize of 24 bits. Fixed with a more careful algebra there. Also suspected similar trouble in EncodeInt24 and applied a scheme of masking so that the function may now be used anywhere within the encoding, not just at its end.

Added the new GUI Word Table editor that logs the entire dictionary. Support was required to get to the common SWord::GetName indirectly through a dictionary pointer. Tried a scheme of spin-locks to mutually exclude SetInfo and GetInfo.

Added 16 reserved words for future use. Added the version field to the serialized image.

}

GUI {

The new WordTableEditor form uses a virtual list box to show a range of words as windowed by the user. Sorting is more difficult and isn't quite wording yet. Double click on a word to load a RichEdit with the information for the word as decoded from the dictionary. Clicking onto any word loads the right-hand-side list box with the words in the grammar that may follow

}


184 Saturday, May 21, 2005

CORE {

Made changes to the memory manager to support the dictionary. Used blocks aren't objects as you know, so not all the fields of AMORPHOUS are defined. Overlapped a few new char fields over the STREE_BASE Object field. The InternalFreeSpace, Checksum, Flags, and LCountSz are new. The memory manager preserves these across MemRealloc calls. These fields are defined by WORD used blocks. Words are used blocks that have references to one another. References are listed in the used space after the header and up to the internal free space tail. LCountSz explains what the bit size of these entries is, and implicitly how many of them there are.

}


185 Monday, May 23, 2005

CORE {

Intense pain to get rid of the BitPos field of SWord and unify with Definition. Found a problem in SetInfo where testing duplicates was comparing information nodes with word nodes which are incompatible. Using Flags to dissociate. Tracked down a corruption on the encoding stream caused by GetInfo which was removing references for contained words. This defect only showed-up after two hours of loading text, obscuring the fact that the input data set contained a duplicate file which triggered this condition and later on lead to a discrepancy between a link index and a word's maximum link count. This was a major setback. With the disappearance of BitPos we can't know the length of an encoding. Consequently, GetInfo cannot skip contained words, and must actually de-convolve their encoding. GetInfo has changed in its high-level aspect. Simplified the ReEncode function to only have one re-base loop.

Removed any unused Dictionary fields. Obsolete Sizes static field.

Changes to TYPE: unused2 renamed to PointerSize to begin an important mechanism of the Morphic Memory Manager which is the notion of address pools. This field reduces by generalization Dictionary::WordCountSize. Renamed TYPE::Phase to SuperHandle.

A major conceptual change: SWord was folded over the AMORPHOUS memory block header. This change makes it possible to obsolete the entire DICTIONARY construct by promoting its logic to TYPE. It follows that any used memory block is potentially a Word, and any morphic memory manager is a dictionary. This version is an intermediary step to that end goal.

Operated a severe re-evaluation of what things are in a dictionary. Each SWord field was questioned and was either obsoleted if it could be computed from other bits of information, or folded over the existing fields of AMORPHOUS TypeInfo if deemed essential to the design. SWord::LCountSz is obsolete because the bit size of LCount can be calculated with a relative few iterations.

We observed the following identities between the fields of SWord and those of AMORPHOUS:

  1. SWord::Index is the TypeInfo.Age. The system isn't really using the Age field yet. The word creation order within a dictionary is the same as the Age sequence. One single problem here is posed by the age of free blocks that disrupt the numeric continuity of the sequence of Indexes. We ignore this subtlety and fold the two fields.
  2. SWord::HitCount is TypeInfo.FlatCount. Words do not have children in the heap. The notion of an instance of a word is quite abstract. Instances of words are the references to that word encoded onto the encoding string. The FlatCount of such instances is the actual hit count for that word.
  3. When we said that the WordTable is actually the cache vector of a dictionary's instances, we had to admit that words too are to some extent type masters over the encoding string. The TypeInfo.Vector of a word is therefore pertinent to hold the beginning of a structure that identifies the encoded reference instances. The word references are a virtual list that may only be traversed using the Decoder, and it is analogous to a type master's chain of instances. SWord called this piece of information BitPos. The notion was obsoleted because BitPos could be derived from SWord::Definition after some decoding. There's a certain similarity between the definition of a word and a type master's instance list in that a word is defined by its instances and nothing much else. The reverse isn't exactly true as there's more to morphic identity than a type master's list of instances. We find that Vector is a short for instances, and word instances is a generalization of morphic instances. Therefore SWord::Definition is folded over TypeInfo.Vector. The identity definition=instances needs to be studied a lot more.
  4. Wanted to escape the SWord::LCount field somehow and came up with the following solution: A used block has a header and a size. Words are used blocks with a special purpose which is to establish a grammar. The used space of a word is all an array of link indexes. The size of the used block itself would indicate the cardinality of its links when taken into calculus together with the pointer size of the address space. The biggest problem here is that the memory manager does not allocate blocks of the exact requested size, because it cannot create free block headers in the tight inter-block spaces. This is a well known matter. Therefore, malloc most often returns a little extra than required. This effect was so far used to the advantage of the morphic dictionary to diminish the Realloc rate, but has now become a problem as this small tail of varying size up to sizeof(AMORPHOUS)-1 makes it impossible to use the block size as the base for calculating the links count. Needed to change the memory manager to account for this extra space with a new field. Had to introduce the new InternalFreeSpce field to TypeInfo. The last thing we wanted here is another field right at the core of the system to shift all files out of alignment. As far as SWord is concerned, it still has a LCount field, only (and this is part of the solution) no longer an integer, but a single byte in size. The free space fragment is less than 36 bytes. Another problem is that the memory manager isn't responding to requests for 0-sized blocks. This is due to the common MemRealloc core that must free those blocks whose size degenerates to 0. We therefore have to create words with size 1, then turn that size into internal free space as discussed above.
  5. We wre left with SWord::Flags, SWord::Checksum, and SWord::LCount fields all 8 bit wide. Needed to find a field in TypeInfo ready to hold them overlapping some other existing field that wouldn't make sense for a word. The current solution is to fold these fields bound in a structure, over the STREE_BASE::Object field which, for AMORPHOUS blocks at least, isn't going to be used. The Object field is only used in containment relationships specific to complex objects.

As a result, SWord has no fields of its own, and is a 0-sized structure (if the compiler allows them). A series of set/get access functions make the conversion of terminology between SWord and AMORPHOUS. Changed the Serializor for dictionary.

}


186 Tuesday, May 31, 2005

CORE {

Reworked some of the TIGHT_MEMORY_CONTAINMENT logic so that free floating objects are owned by their allocator. CreateInstance binds objects in this way. INIT doesn't. Want to get one step closer to portals and generalized persistence.

Changed the STREE::Serialize routine so to reverse the order of children loaded from persistence. Still got to keep this logic under observation.

Added a fault test to K5CORE to exercise the new exception handler built into Core.

We're working on an exception handler that can take control over to the dis-assembler, emulate the faulting instruction and return to faulting code after the fault point. We need this to virtualize memory and achieve de-fragmentation.

Enhanced the DICTIONARY with sound capabilities. The alphabet has associated waves in the resources of the Core. The GetInfo routine has a new sgAudio mode where it plays the waves of letters, words, and ultimately text.

}

BASE {

Added the new HOST_OS_AUDIO class to abstract the sound subsystem. Only one function is currently defined: SoundPlay.

Added the ExceptionFilter function to analyze an exception code, dis-assemble the fault origin, and try to skip past the faulting instruction. The TDisAsmBase_FillText emulator is ported from ANNA K4, the first assembler module brought from K4 into K5.

A bit further now. I seem to have achieved a remarkable result: ANNAk5 is calling logic from ANNAk4. Indeed I've changed the makefiles of K4 to build the ANNA.DLL flat 32 version. Was able to link it into BASE.DLL and call the dis-assembler across the two.

}

GUI {

Fixed a problem with loading something into Gaius5. The thread canceling condition was wrong.

Added sound capabilities to Dictionary. The Word Table Editor has a new button that initiates the Core's reading decoder.

I can't truly evaluate the future potential of this speech engine that emerged here with the dictionaries. We'll see.

}


187 Monday, June 06, 2005

GAIUS5

{

Gaius 5 is now focusing the master scope browser when clicking on the I/O links. Gaius has to indirectly call the FindExpand of the Scope Browser. A new flag to enable the sending of the master browser refocus message from the IO panels. Generalized the Compile method so that the IO panels can also use it to show info at a higher depth level. GetNearestReference looks backward in the current concept for a heading and focuses it in the master browser. A correction to the whole word text search. Added hints to the IO browsers that show the full scope of the selected link.

}


188 Friday, July 08, 2005

CORE {

Found and fixed a subtle problem in the memory manager leading to corrupted block content when a used block is relocated such so its original location is at the end of a free block with which it gets concatenated. In this scenario, the original location pointer was wrong to be the resulting concatenated block.

Fond several problems in the DICTIONARY:

The bit masks used by Encode/DecodeInt24 were clashing with some other vector in the system by the same name, and getting those values. We were not persisting enough of the length of a word, and had to also consider the internal fragment. This little piece of information is used in calculating the actual count of links, and the MMM must not be allowed to change it without the dictionary knowing. Trouble with large dictionary files which must now be loaded in chunks to avoid API errors. At present, there's justno way to obtain a coherent persistent dictionary when we operate anti-duplication maneuvers. Had to disable the duplicate removal logic.

Moved GetLCountSz to the header-file so that the GUI can walk the word table too.

Found some stupid mistake in the WordTable editor for the dictionary.

}


189 Tuesday, August 30, 2005

BASE {

Our threads are now spawned with try and __except to handle exceptions. beginthread was needed to establish the exception handler stack frame. Added HOST_OS_RT::FileTruncate. Added the HOST_OS_GDI::GetCurrentObject, the key to resolving that handle leakage problem. Protecting SelectObject and DeleteObject against null arguments.

AGING_TREES is not defined. Age isn't present in the system as that would lead to a shift in the persistence. Added the STREE_BASE::Index instead.

The DNA_ENTRY is a new structure made of Type and VMT pointer. It makes the junction between the static executable (code) and the re-locatable data. OBJECT_BASE::Types renamed to Dna. Please see Concept Report for a description of what DNA is.

AMORPHOUS::IsObject now returns the DNA entry for that object. The DNA pointer is stored as the first entry of the VMT called the __dna__() non-function. AMORPHOUS::Activate is a new virtual vector for activation/inactivation of objects during relocation

Added the new SProtectionInfo class that holds the limits and the current state of a relocation operation. Please see ANNA K5's white paper on the matter of Morphic Relocation. AMORPHOUS::SetTypeMaster is now common as it's prototype cannot match that of OBJECT::SetTypeMaster.

}

CORE {

This version of the Morphic Core is self-aware. This is the most important result of this development yet.

DNA is a vector of (Type,VMT) pairs in the order prescribed by UIDS.H Its role is to link a Morphic Relocatable Enterprise structure to module of code. The morphic system's DNA is defined in TESTS.CPP, which is the startup module. All CLASS::Type static pointers are now instantiated in the order prescribed by the class IDs declared in UIDS.H, in TEST.CPP to form the system's DNA vector of (Type,VMT) pairs.

TSuperKernel constructs a MORPH object over a char buffer to avoid some C++ effects and defects.

The morphic core starts-up differently now. The BSS segment of the core is Morphic Data now. When a persistence CoreModuleName.BSS file is found at startup, it will be read and mapped into the BSS. Otherwise the core is constructed from scratch. The BSS is saved to this file on exit. The process can be disabled from the CAT_MORPH::ofNoPersist flag. Morphic persistence is a complex issue that involves relocation. The host operating system is free to load the Core DLL at various addresses, so the BSS must be relocated. The core is an auto-morphic structure now, one that intimately knows its internal structure and can shift in memory as instructed by the loader. The rebasing operation requires no relocation table, which is a major feature of the Morphic Operating System over the classic systems Relocation in the morphic space is the act of copying the substance from one medium with some structure to another medium of another structure, fix the internals of the object to reflect the new location (re-base), then activate at the new location. This process requires a self-awareness pass over the object. The procedure consists of getting the type of the object, and de-convolving it to elementary types, mapping the offsets over the object to locate them, fixing them one by one in a sequential and highly ordered manner to reflect the new location (relocation must not occur twice at the same place as driven by the overlapping regions of union types). Of interst in relocation are only pointers and of them only the so called inbound pointers. These are places within an object's substance that link to other places within the same object. Moving an object must update all the inbound pointers and leave the outbound pointers as they were. One remarkable pointer is the virtual method table of any OBJECT aspect of an object. Objects are, as you know, solids made of other objects by containment.

The virtual method pointer is neither inbound, absolute, nor outbound. It is an abstract pointer. Abstract pointers must be encoded when persisting and decoded when loading a morphic structure. The process of walking an object map to encode its abstract pointers is called Inactivation, and the converse of decoding is called Activation. During an Inactivation pass, the object's VMTs are unlinked and replaced with DNA ids. Conversely, the DNA ids are replaced with actual VMT pointers upon loading.

Instrumental in the process described above is TYPE::GetSize. GetSize is the fundamental algorithm that walks the format of an object adding sub-sizes to the overall structure size. The walk over the containment relations is done through TYPE::GetLasRELATION and TYPE::GetPrev. Besides its primary role in calculating the size of a structure, this iterator can also be instructed to issue a callback upon each step to list the object. This result is speculated by the new DebugDump routine and the GUI TypeInfo viewer. GetSize may be passed a SMapFormat structure to include the ProtectionInfo. Protection is a structure that defines the range, direction, and current state of a relocation process taking place in a medium. The information is passed through all the iterators, so as to know the range of protected and unprotected pointers. The dref call is used to safely dereference a potentially relocated and therefore protected pointer, Protect, and Unprotect are doing the actual shift in the persistence for any inbound pointers as decided in the main GetSize algorithm after inspecting the type, kind, and sense of each relation. The default static int CbMapFormat (SMapFormatCB* fmt) supplied by the Core logs a line to the debug output.

Morphic self-awareness is ensured now through-out the system by getting the categories to multiply descend from RELATION as many times as intra-pointers the category has. The matter is complex and requires more discussions. Morphology isn't an easy thing.

The new TYPE::SerializeHeap uses morphic self-awarness to store and load a range of objects. When storing, it first deactivates the entire heap by walking it. Since objects refer to one another within the heap, the ProtectionInfo of this operation must cover the entire heap, not just the current object. Then the objects are streamed to file lead by some size information. Loading the heap is the converse, and a GetSize call is issued then wanting its relocation side effect.

I'm pursuing the Generalized Persistence Mechanism, which is now possible. A deactivated object can be aged-out of the heap providing that all the relatives and the environment were informed and they adapted their outbound pointers. An object may now be de-activated while protecting the back links of the environment. OurExceptionHandler was introduced a while back ready to catch memory violations, inspect and fix them. It can now be enhanced to bring Inactive objects beck from a deeper level of persistence into the main heap and activate them.

A new formalism of relations

{

The following table describes all fundamental binary relations of the general form

S.r:A*B where r is a relation in scope S from A to B.

--+-----------+------------------------------+-----+-----+------+

Nr| Relation | Name | r | A | S |

--+-----------+------------------------------+-----+-----+------+

1| S.S:S*S | Null | - |

2| S.S:S*A | Auto-definition | - | +A |

3| S.S:A*B | Definition | +B | - |

4| S.S:A*S | Will | +S | - |

5| S.r:A*B | Containment | - | +B | +r |

6| S.r:A*S | Contract | - | +r | +r |

7| S.r:A*r | FROM auto-relation | - | +r | +r |

8| S.r:A*A | Fractal | - | +{A}| +r |

9| S.r:S*A | Has | - | - | +r+A |

10| S.r:r*A | TO auto-relation | +A | na | +r+A |

11| S.r:r*r | Auto-fractal | +{r} | +r |

12| S.r:S*S | Promise | - | +r |

--+-----------+------------------------------+-----+-----+------+

There are several types of relations in the morphic system as follows:

1. S.S:S->S is the null auto-relation. Since this relation is of its own scope

it defines the beginning of an independent morphic entity, a cell, or a clone.

All morphic systems start off as null auto-relations. This relation also has

the properties of the (4) registration relation.

2. S.S:S*A Scope TO auto-definition induces A into S.

3. S.S:A*B Is a scope definition.

4. S.S:A*S Scope FROM auto-registration induces S into A.

5. S.r:A*B R is a relation in scope S from type A to type B. This kind of a

relation is the most general case that describes an induction of B space

into the substance of A under the control of S. Scope S owns and dials the relation.

6. S.r:A*S R is a relation in scope S from type A to type S.

S owns and dials the relation, allocates space for it.

S induces space into A. R is a "registration" relation.

As more such relations S.Rn:N*S are born in the system, S grows to hold the

registration of each relation Rn. With each new relation added, all previous

N-1 types that depend on S also grow. These registered types know one another.

7. S.r:A*r R is a FROM auto relation in scope S from type A. This kind of a relation

describes an induction of R space into the substance of type A and of A into S.

This relation is an inclusion of A into S. R is self described.

S owns, dials and allocates the relation.

8. S.r:A*A Self-relation much like STREE{ STREE*STREE.Next }

R Induces a fractal upon A, and as such it must be depth limited.

Scope S owns,allocates, and dials the relation.

9. S.r:S*A R is a relation in scope S from type S to type A. This kind is the most common to users.

Type S owns and dials the relation, and allocates space for its inprint.

These are relations of "free-will".

10. S.r:r*A R is a TO auto relation in scope S to type A. This kind of a relation

describes an induction of A space into the substance of the relation itself.

This relation is an inclusion of A into r and of r into S under the control of S.

R is self described. S owns and dials the relation, and alocates space for it.

11. S.r:r*r R is a fractal auto relation in scope S. This kind of a relation

describes an induction of R6 space into the substance of type R and of R into S.

This relation is an inclusion of R into S. R is recursively self described.

S owns, dials and allocates the relation.

R is recursive and should therefore have previous or following members, as well as a

determined fractal depth count.

If R doesn't have previous or following members, then it is void.

This is the true nature of the Void type in the Morphic system.

12. S.r:S*S Scope self-registration adds r to scope S for dialing.

}

A type iterates its relations during a self-awareness and identity mapping process. A type's identity is determined by the strongly ordered chain of relations that impact on its size. Allocation and identity are semantically identical issues. A type identity traversal should ignore any relations that do not impact on the type's size.

INVOKE_SYSTEM_STRUCTURES is not used by the core anymore, just by the fax system.

ISTYPE just declares the Inherits typedef, and obsoletes the virtuals IsType, GetClassType, and GetDirectAncestor which are now answered by the new Type awareness mechanisms. IsType, however continues to be called by the GUI and therefore must be virtual. It loops through the relations of the object's type to find the inheritance relation and then climbs the ancestor list to find a match. This process is type-info sensitive.

The SET_TYPEMASTER macro isn't doing any VMT manipulation. The messages of SelfTest are more careful with invalid names to crash less. UPDATE_PROXIES_OLD wasn't used. VMTEntryType promoted to BASE. vmtSetTypeMaster is a registered static virtual called by CreateInstance. CreateNewInstance is obsolete. The TypeMastersVector moved from TYPE to OBJECT. Another Find flag: FIND_COMPLETE used internally upon the secondary recursion levels of FindCreate to keep a search path coalescent. Changed so that FindCreate doesnt use the heap for its internals.

TREE::IsLineEndpoint is now common.

OBJECT::DONE just does SetTypeMaster which is the actual destructor.

More integrity tests: OBJECT checks to see if it owns its name and information blocks. It's an error not to own these blocks for free floating objects. Contained objects such as the connector of a relation point to the name and Information strings owned by the relation.

The principles of creating a default instance of a type have now changed. CreateInstance is a static virtual.

OBJECT::DebugDump now produces the Morphic Listing of the object.

MemReclaim was horribly slow due to always rolling back the instance pinter. Now we're scanning forward past all objects (involves calculating sizes) to find the first used/free block.

Fund and fixed a problem with not updating the Instances pointer to the latest maximal free block as required by the Map Viewer.

MemGrow now has the logic of DICTIONARY. Found problems with the MMM spinlock.

Introduced the BASE category (ID 10). Added bitmap resources to it, thus affecting the IDE project.

BASE holds the following type masters: Foreign (11) Void , Bool, Char, Int, STREE, AMORPHOUS, OBJECT_BASE (18)

The ID of ABSTRACT was changed to 20. Added point to ABSTRACT.

Automated some aspects of initializing the Core, so more IDs and fields had to be renamed.

ABSTRACT doesn't descend from USER anymore when it contains USER. It gets the IP and Port from there. ABSTRACT is a vector RELATION now.

No more C++ destructors and default constructors.. There is just one construction/destruction place in the C++ sense at the level of TEST.CPP during core startup/shutdown. It is solely responsible with locating the VMTs and registering them in the system's DNA. The new core builds with external virtual method tables, and TEST.CPP instantiates all of them.

Operated an overall cleanup of the model for everything that was polluting the scope.

Ancestry as defined by the new GetDirectAncestor method is generalized to use the self-awareness mechanisms, and does not rely upon coded information.

TYPE::Active (SProtectionInfo* pi) can be called with 0 to return the activation status, or with a configured protection range to either activate or inactivate the given object aspect.

TYPE:: GetInstanceSize returns the MaxSize field as configured by each type master. RELATION for instance generates nodes.

RELATION must not override TYPE::GetSize and to calculate the size of its field we now have GetFieldSize

RELATION sets the same name and information for both field and connector.

RELATION::IsContainment (RELATION::ERelationType rt) looks to see the cases where the relation impacts on the size of its endpoint type.

TYPE::GetInstance (int index) is new and used by the Dialer mechanism that now shows the name of the selector whenever possible instead of its index number.

The DefaultAction of a STREAM is to release its Interlock structure.

Must build the core in such a model where enums occupy minimum representation space.

NODE and LINE

{

There is a new line endpoint decoration corresponding to relations from and to themselves consisting of a circle and a line separator.

Former mallocated blocks were owned by the allocator. Now their ownership is claimed by the object pointing to them.

NODE and LINE override the Active call to rebase their Font and BendPoints pointers. This might now be a mistake in light of the new morphism algorithm that relocates all inbound pointers.

A new NODE shape shPoints was introduced, as well as a NODE::GetPoints algorithm to collect the contour of a node as recursively defined by its children. We need this mode to create maps. Algorithm is driven from SerializeFace.

LINE overrides the set_Name and set_Information methods to route them to the container Relation if necessary.

Enhanced the NODE::SelfTest with a check to see if the node owns its font. Likewise the LINE enhanced its SelfTest to see if it owns its BendPoints.

Found a problem in AutoBind where the ofAutobindDisabled wasn't taken into account and a cable was constructed when it shouldn't have, especially when the system was shutting down and no new memory allocations were allowed.

Node removal needs/must not involve the AutoBind mechanism when the node is orphan.

There's a new NODE option flag to take of the display of labels globally.

point::HitTestPoly now returns the segment index on the contour which intersects with the test segment.

point::Between(A,B,t) returns a point at a parametric position t between points A and B.

point::WeightCenter (A, B, C) returns the weight center of a triangle generating a 1/2 A' point on [BC] and a 2/3 point G on [AA'].

rect::SetDegenerate, InflateTo, GetCenter, are all used now in the new tiling mode tsCenterAroundChildren.

}

Adapting the category to the new principles of the morphic core whereby the category defines a Relations vector to map over its contained fields. This is to compensate the lack of vector inheritance in C++.

Ample corrections to the canvas serialization to not consume handles. Had to GetCurrentObject and delete it. TCanvasStream::SetDevice must delete the old handles and take note of the new ones. The stream gets an image of the very first pen, brush, or font handle that gets installed.

DxD::ClearScene is done only if the canvas isn't locked.

FSYS: Renamed TDatabase to DATABASE to avoid a name clash. Added the DATABASE type master to the category.

INSTSERV: Host, etc needn't be morphic pivots, just hierarchies.

COMPILE: A new mapping relation called Grammars. Severe changes to initialization, and destruction in light of the new requirements of the self-aware morphic core.

ROSE {

Our scope selector is ":" not "::" like C++. Factorised al FindCreate calls for better control. Enums and methods flag the generated type with one of flEnum, flMethod or otherwise the default flNormal. Reversed the order of constants in an enumeration. When encountering a new type the loading file, we create a type master that is equivalent to TRoseBaseObject and needs not overwrite the VMT since the new CreateInstance takes care of that. New binding to Int, Void, Char and Bool which now register with the DNA, and belong to BASE rather than ABSTRACT. Reversed the order of insertion in the lists so that our scope is in the same order as that declared in the loaded object files. The default action of ROSE is now to clear all its loaded type masters.

}}

GUI {

TypeInfo {

Major enhancements here with the assistance of the new DNA map of the Core. TYPE::GetSize can now be programmed to issue a callback to de-convolve a type and map a format over a set of instances of that type. This is a milestone of this development.

The new viewer fills a grid within a window range with formats and values.

In its columns it shows: GapOverlap, Offset, FieldSize, Relation Kind, Type, Containment, Cardinality, FieldName, followed by one data column per instance checked in the instance list.

The status bar is a series of sizeable panels showing global info for the structure: Size, total gaps and total overlaps, and the number of formats.

}

SE {

CenterAroundChildren, Animate, Edit_Zoom, Cone senlection and unselection.

A new TYPE field bund into Flags to further qualify a type master as Normal, Enum, or Method.

A new tiling mode tsCenterAroundChildren.

The default action is launched on a separate thread.

Introduced a node Animate action that spins the selected node around. This tests the speed and handle leakage of the system.

Configuring the static options of RELATION, and CAT_MORPH.

Semaphored InMouseDown and InMouseMove.

A new NODE face to tightly wrap around the contour of children.

}

Gaius5 {

A new generation mode to emit the Morphic Script.

A new generation mode to emit a series of macros that may be included back into the Core source-code.

}}

K5CORE {

New console commands to dump core format, toggle core persistence on/off, create and destroy a clone of MORPH. The K5 console now defers to another command interpreter inside MORPH.

}


190 Tuesday, September 06, 2005

BASE {

An assembler BASE module for the virtual pointer protection mechanism.

}

CORE {

Caught another problem in the heap serializor. The amorphous handle must never be virtualized, and must be restored if it was ever changed by the protection mechanism. Segregated the protection mechanism to its own module so to turn it to assembler.

Enhanced the TYPE::GetSize mechanism to deal with the Numbers Theory, which says that a type is the size of its largest constant. So fundamental types that have no base relations and have constants instead are now calculating their size from these primitives. The Energy enterprise file is an example of a structure started from scratch by defining int, bool and real using constants.

Added a generic destructor that iterates the fields of a type to find strings and frees them. All name and information strings are now owned by their respective objects. Deleting all objects leaves the heap clean now.

}


191 Thursday, September 08, 2005

CORE {

Rework of the string allocation in the double compiler for petal files. All strings are now owned by the loaded type masters.

The file format allows a value field to either be a number, a boolean, or a string. There is no way to tell them apart. If a type master first found a string value, it cannot then retype that field to int or bool. The new logic sets boolean and ints in string literal format if value was first set to be of string type. All this is for the DefaultAction cleanup to work. The order of cleanup is from most complex objects down to the most basic ones.

There were problems with the heap serializor and with calculating the size of a structure. In short, an enterprise file serializes with no regard for memory order. Its main concern is to follow the coherence of a cluster's inner linkage. Therefore, a scenario has arisen where a type master was loaded at a higher address than some of its instances. Another problem with the serializor is that it doesn't preserve scope order, which is very bad. The Energy cables.BIN was defining some types and operators that have nothing to do with the Core, yet they are represented by nodes, lines, etc. Did a rethink of what size should mean and resolved that a type has a header type and a data type. The two are not always the same. An object instance occupies the sum of both sizes if the types are different. To persist morphically such an enterprise, all its embedded types have to be coherent. A structure is coherent if it is contiguous and doesn't have overlapped regions. A new THierarch::DefaultAction method was coded to fix-up the coherence of a structure and reorder the list of instances according to what we've just discussed. GetSize had to be enhanced to calculate the maximum constant size by iterating the node instances of the given type master. This is very delicate.

This version of the Morphic Core is able to persist independent type pools.

Enhanced OBJECT::DebugDump to return an error code of 1 if there are gaps, and 0 if there are overlaps. This should be another DNA integrity test for type masters.

}


192 Friday, September 09, 2005

BASE {

We've layered-out the base micro kernel in such a way so that the Core may initialize it as an OBJECT.

The uKernel is now the ultimate owner of the proxy and of the Interlock scopes.

}

CORE {

We've changed the way the Morphic Core initializes. It opens the BSS file, constructs, then closes the file. On exit it opens the file, destructs the core, then closes the file. This way, the file can be shared. Also, the large BSS data space was slow to map. We're now asking the inherited memory manager for a large chunk of memory before installing our morphic memory manager. We release that block of foreign memory after restoring the memory allocation vectors to their original values.

When the Core constructs, it issues a functional morphism by claiming the identity of the micro Kernel as OBJECT. It deregisters the uKernel just before shutting down the type master space. Changed the heap serializor to lead the file with the count of objects. This determines more precisely the length of the re-mapping loops.

}


193 Sunday, September 11, 2005

BASE {

Passing some flags into the ExceptionFilter to instruct it to ignore or disassemble the faulting code.

Added Options to OBJECT_BASE to drive the exception handling process. The console tester toggles these flags before and after the threading test which also involves the exception handler.

Felt that a potential problem could be caused by reentry to Fork while the spin-lock is busy. Preventing.

}

CORE {

Fought a long standing battle to re-align the Morphic Core so that all objects have a handle to the outer world. The field had to be other than the TypeInfo Children which is desperately needed to hold the hierarchical structure of used blocks and so forth. Now they do. We have re-assigned the OBJECT::Semaphore field to the new OBJECT::Handle field. This change required a total re-work of the Interlocking system to not use system semaphores anymore. Our objects ARE now the semaphores and the threads locking onto them now get suspended and resumed. Once the Semaphore field turned into Handle, it could be promoted up the hierarchy all the way up to the AMORPHOUS type while preserving file system compatibility. The ptrSemaphore of the ABSTRACT category has moved to BASE::ptrHandle.

The new Interlocking mechanism.

Struggling to find a way to suspend a thread that has a GUI message queue to keep alive. Threads induced a new STREAM::flBlocked flag, and the new Suspend and Resume methods. Suspend sets the flBlocked flag, then loops peeking and dispatching messages as long as the flag is on. Resume resets the flag. Initializing threads have to call the THREAD::Attach method to interlock the new thread to the MicroKernel. GetCommonParent of TREE was promoted to STREE so every hierarchy can benefit from it. The Interlock is now using it as well to see if two objects are interlocking or not, or whether they are locked by different threads. The new interlock only does a basic subset of what the older one did, and is still under development. It works as follows.

1. Two unlocked objects may simply interlock by inserting one into the other's Interlock scope.

2. A morphic thread may still run GUI messages when suspended. Locking will not be permitted for such threads.

3. Locking onto a null owner means locking onto the current thread.

4. Locking onto oneself is not permitted.

5. If the locking source and owner have a common interlocking parent, then the parent may either be the MicroKernel or another object. Only threads are allowed to interlock with the MicroKernel. The MicroKernel as the common parent means that the two locking objects belong to different threads. The source object is already locked by another thread. The calling thread must therefore interlock with the source object and suspend its execution until the other thread releases the source object. When the source object is eventually released, its interlock children are searched for a thread that can be resumed. Resuming the thread would bring the logic to the following common scenario of an unlocked source object.

6. If the source object is not locked then it is inserted into the owner's interlock and the Lock method is successful.

7. If the two locking objects have a common parent that is not a thread, then we are performing an over-lock. An over-lock is an object locking another object all locked within the same thread. This scenario is non-blocking and must not suspend the current thread. The level of the overlock must however be noted. In this case we increment the source object's Interlock flat count each time an overlock occurs. The Unlock method must not break the interlock relationships unless the overlock count has reached equilibrium with the actual number of children of the interlock. It's only when the overlock is in equilibrium that pending threads may be resumed.

8. Any object locked by the MicroKernel is considered to be unlocked.

In short, the Interlock is a sophisticated piece of real-time logic for us rather than for you to worry about.

STREE::GetFrontier collects the finger-tips of the interlock tree, and is used in the Schematic Editor. The problem here, however, is that the last locks should be collected first, allowing the user to reselect an object that now has threads pending on it as selected by other users of the same structure. We've reversed the order of objects returned in the vector of finger tips.

The current file system format proved to be highly resilient, and the struggle to not break it, so very hard as it my have been, was indeed worth wile. We had to think everything through on paper before introducing any new object at the top of our scopes and this forced us to discover many semantic equivalences between algorithms and structures. Right now we need this AMORPHOUS::Handle field to link proxies to core objects back to user interface objects such as windows and tree nodes.

}


194 Tuesday, September 13, 2005

BASE {

All GUI objects are now object proxies. They attach to the target to monitor it, then they close when the target is deleted. The TNotifyRelation is therefore an OBJECT_BASE descendent so that it may be a proxy, and keeps the former GUI activation tree scope. A C++ constructor attaches the new TNotifyRelation to the target passed as argument.

The base micro kernel has to be partially initialized in advance of loading the Core. GUI proxies must be allowed to attach to the micro kernel and the Proxies chain must not be trashed later by initializing the Core. Introduced a default constructor of OBJECT_BASE to take care of initialization of the static uKernel.

}

CORE {

The partial initialization of the base micro kernel must not be trashed by re-initializing it. Well, this is an application of one of the most important issues that the Morphic Core aims to resolve: Promotion. Object instances of the Morphic System are never quite finished and their construction is continuing. Object instances grow as they perform, and reduce as they evolve. The base micro kernel starts off as an OBJECT_BASE, but must turn into a NODE when the Core initializes. This is so that the Core may instantiate multiple cells and register them in the kernel's scope, or to establish the Interlocking system which is specific to the morphic core. See ANNA K5's white paper for details.

SetTypeMaster must now consider that if the Interlock or Proxy trees are owned by the current object, then they were partly initialized as OBJECT_BASE already, and preserve these trees.

Found that the Unlock logic was not respecting the order in which threads locked themselves onto a resource. Had to reverse the the order of parsing the children list of an unlocking object.

Toggled the dictionary's FAST_ENCODE_DECODE build option back on. Declaring some low level functions as __fastcall. Removed any unused field. We're aiming to merge DICTIONARY and TYPE. We don't know what to do yet with the MemoryBlock field.

Found a problem in the serializor with ownership of strings. When reading from an enterprise file each object must link the loaded string to itself. OBJECT::set_Name does not reallocate the string if it equals the argument, thus avoiding orphan used blocks during serialization. Also, the function can (and should) be used to bind the string to this object. Same applies to set_Information. A relation must optimize its memory allocation by resetting its name and information when they're the same with Connector. RELATION::set_Name/Information must free the connector's string if one exists.

}

GUI {

All GUI forms are now object proxies and they get destroyed when the target object is deleted. All GUI relations no promote a target object pointer upon which they have to lock. The former Navigator Config.CPP/H held the definition of a Null relation, a TopRelation, and a TForm_Relation. Moved all those to the top of the GUI system to be shared by everyone as components of the K5Components. There were many problems with this "simple" move, as the TForm_Relation had to be registered, or else the application constructor failed on the VMT lookups. Config.H/CPP were renamed to Components\UiProxy.H/CPP. Factorising the TForm_Relation to the K5Components simplified the build of all GUI modules which need not include config.cpp now. Took this opportunity to re-think and clean-up those BPR files that are so easily trashed by CBuilder leading to inoperable precompiled header files, and others. Kept a config.h inclusion gate though.

}


195 Thursday, September 15, 2005

CORE {

TYPE::PointerSize is char not boolean.

Added another relation flag flAmorphous. Amorphous relations are hidden away from the TypeInfo mapping, they are not subject to morphic move, their size is calculated directly as sizeof(target)*relation_cardinality. Their purpose is to reserve extra space for a type. Dictionaries currently use such a relation to define the size of their heap.

The list of relation names of ABSTRACT grew and had to be split over 2 enumerations.

The DICTIONARY::MemoryBlock field was obsoleted with a superior instantiation technique whereby a DICTIONARY is a TYPE with an ExtraData relation that may be configured. The heap space of any type master is calculated between the limits of the header and the end of its morphic map as returned from GetSize. Any type master can potentially be a memory manager that owns a heap. Heap space can be reserved at runtime by editing the ExtraSize relation.

Found a problem in the MemGrow algorithm that maniphested itself at the end of a cycle through all blocks, failing to resolve the end condition. The stepping rule is to follow the allocation chain until memory is made available or the initial free block is found in full circle. However, a used block between two free blocks may have been de-fragmented to the beginning of the union of these three, and the memory manager cleared the header of the right-hand-side free block. This changes the stepping rule making the end condition impossible to catch. This scenario is now tested explicitly to correctly exit the MemGrow loop. I must have felt that there was a problem here when I got the MemRealloc to zero-out the merged free block header.

Changed the way we handle profile information blocks. These blocks are now attached to TYPE descendants from proxy GUI interfaces. A local type editor installs a profile information block into the target object, replacing the existing block. The GUI uses the set_ProfileInfo method to do so and swap the profile context. The incoming proxy copies the profile data of the outgoing one, like a kind of a dispatcher. This TYPE::ProxyData field should really generalize at the OBJECT level and begin to mean UserData, where a client of the Morphic Core can attach any tag. We're now saying that profile information is half in, half-out of the morphic core, an object data extension for which the core doesn't allocate space unless it is told to, but can handle it if it exists.

The size of an empty type is defined by the maximum constant of that type. Iterating the constants must be done within a spinlock protected block, so that the memory manager is prevented from updating the list.

Renamed SWord to TOKEN. A token is a re-mapping of the AMORPHOUS base class. It adds cross-reference functionality and has new manning. A morphic memory manager manages the following types of constructs: used blocks, free blocks, tokens, objects. A token is a special kind of a used block the main difference being that used blocks are exported to users of the memory managers, and therefore link back to the memory manager that allocated them, while tokens are internal to the memory manager and the user must use the dictionary entry points to make sense of tokens. Directly accessing tokens is not possible. Tokens have special features, in that they cross-reference, and they leave a trail in an encoding string, if the memory manager has one. Tokens are the foundation of any parser. Our language interface is dictionary based. A token can introduce new fields, but currently doesn't as it reuses those of AMORPHOUS. The dictionary has no proper fields of its own other than the Flags of the parser. All other fields have folded over existing inherited ones. The WordTable is TypeInfo.Vector. WordCount is the first entry of the WordTable, making the word table essentially a String. The EncodingString is held in TYPE::SuperHandle, and its length is prepended.

A significant achievement is that dictionaries have now generalized their MemGrow with TYPE, bringing us a step closer to a base memory manager that can de-fragment its content.

Serializing the dictionary is currently under re-evaluation and was disabled. It first forces a de-fragmentation by allocating the exact space of the heap, a condition that gets handled silently by the MMM without throwing an error.

}

GUI : Found a problem with the auto-tracking logic of the memory map viewer which wasn't getting the cache pointer correctly. Changed the handling of Profile Information blocks, which are now owned by the local type editors and dispatched in and out of the target object.


196 Monday, September 19, 2005

BASE {

Changed the layout of STREE so that a dictionary may ignore a token's VMT, Parent, Prev, and Next fields.

OBJECT_BASE starts-off the profile interface by defining the CurrentSample vector. Added the GetStatus virtual; method.

}

CORE {

Hunted down that encoding corruption. Turned all bit-stream positions from int to unsigned. Link indexes are also unsigned now.

Added new profile vectors for dictionaries: TotalLinksIn is the number of lines wrapped around the dictionary, used to calculate the new GradientLinks which is the trend of new links per thousand tried. Added the GradientWords, and the new WordsPerSecond which shows the speed of coiling operations. Added the Duplicates count and an IndexEncodingError. The new principle is that no TYPE contains the ProfileInfo, rather points to it, and one such structure must be supplied by the local editor. However, the dictionary wants to store those in the heap file, so a global profile block must be made available for cases when no GUI is attached such as is the case when loading the dictionary.

There was a serious remanoring of the dictionary serializor which now defragments the heap first, then stores each token in one atomic exchange starting from a token's first meaningful field, AMORPHOUS::TypeInfo.Children, ignoring VMT, Parent, Prev and Next. The storing loop breaks out when it finds the contiguous free block, and resumes from its adjacent following block. Loading on the other hand is even faster as the dictionary flood loads the entire file towards the end of the available heap space, and then moves each token to the front and re-creates its AMORPHOUS header.

GetWordName is not a function of TOKEN but of the DICTIONARY. This is in tune with what we declared earlier that a token must be accessed through the dictionary functional interface.

Re-worked the InWrite and InRead dictionary spinlocks, and placed them at the level of DeEncode. Got the SetInfo to call GetInfo directly bypassing and not recursing the spinlocks set in DeEncode. Same for GetInfo.

Traced and corrected some string ownership problems in the XML parser.

Initialization of the grammar is now done once by the parser's entry point when the dictionary is created.

Found a horrible problem, I don't know how we survived it for so long, probably responsible for the index corruption in the encoding string. The problem was in DeleteWord, where the backward shift length was negative.

Decode/EncodeInt24 is now unsigned, for a more reliable algebra.

Added a few GetStatus methods for MMM and DICTIONARY to fill with some information about their spinlocks.

This version of the dictionary feels a bit faster, but might just be my sense of boredom with it all.

Corrected the layout of THREAD. Turning its Options field to static.

A nasty change to CreateInstance. With our new concept of heaps, we also have to take their size into account when constructing an instance or when hopping over one during a memory scan. A new comparison between the values returned by a type's GetSize and GetInstanceSize is done and if actual size is larger than the DNA size, then it would be preferred. We will soon have to rethink this and bring it into perspective with MaxSize and the notion of Constants.

We're not clearing the collating free block headers. Found and fixed another problem in the defragmentor MemGrow when the wraparound condition occurs and collation also happened on the initial/final free block. Got to do defragmentation again to complete a full circle and make sure that only one free block is left in the heap. The former version ended up with two.

}

GUI {

The abstract TForm_EditFrame calls OBJECT_BASE::GetStatus on every RefreshTimer to show something in the status bar of the local editor.

Added the PrintPreview and Save actions to the Profile Editor. Printing a chart and saving it enhanced the quality of the system.

Auto-sizing the grid columns. More grid columns corresponding to a new mode of operation: there's a Y multiplier field and an X redirection field for each profile. The X redirection field is 0 by default and instructs the chart evaluator to use the value of the '0' profile, which is now the CurrentSample, as the X reference. The user can now change the X reference of a series by changing this value to another profile index. Another mode of operation is the &number in the X redirection field. This construct evaluates to the actual constant number and is used by the new grid evaluator to update the Y value of that X series and produce bars.

All changes in the grid may transit invalid states, and must therefore be tied and caught. A better way to clear the content of a series.

Enhanced the WordTable editor with an Order vector. This is an indirect vector of indexes to the dictionary's real WordTable. The Order vector is 1000 granular, and larger then the actual WordTable. There's a new Sort method that orders this indirection vector according to the CurrentColumn and Reverse settings. CompareWords returns an integer, the difference of the two comparing items, and can be negative, 0, or positive like a strcmp. Sort must interlock the dictionary to prevent updates when contending with SetInfo operations. Well, it works now, but either there's something wrong with my sorting algorithm, which theoretically should go through a maximum of n(n-1)/2 iterations, as you know, or comparing names is excruciatingly slow. In this latter case, we must immediately promote the sorting logic to the dictionary class and get it to do grammar-base sorting.

Fixed the side effect of selecting in the ListBox to bring a related word into focus in ListView.

}


197 Saturday, October 01, 2005

BASE {

A severe change to the registration mechanisms. The DNA vector is a 256 -long array allocated by the Base Micro Kernel, and not by the Abstract Core as was the case before. This change allows other core modules to also register their DNA and coexist.

GetTypeID is now globally functional and returns the ID of an object's type master. Installable core modules do not need to override this method anymore, but they do have to register their type masters.

Found and fixed a problem in the INI file loader where the hex-style of storing type IDs was confusing above 9.

}

CORE {

A severe change to the registration mechanisms. The DNA vector is a 256 -long array allocated by the Base Micro Kernel, and not by the Morphic Core as was the case before. This change allows other core modules to also register their DNA and coexist.

The Type static members are now declared by the ISTYPE macro, and were removed from the model. ISTYPE takes another argument, the EquivalenceClass, which is the type of the type master pivot being declared.

A core module first establishes a private DNA vector (a genetic fragment) then copies it to the base micro kernel.

Care needs to be exerted when morphing type masters to also inform the global DNA vector of such relocations.

Found a mistake in the ancestors of CAT_BASE and CAT_MORPH.

INVOKE_SYSTEM_STRUCTURES is obsolete. All core modules must initialize their category multi-relation.

OBJECT::GetVMT is obsolete. The new DNA system is better.

Found that getting the time from the ANSI library is a thoroughly painful exercise of divisions, and multiplications. We avoid doing it in the MemRealloc and MemReclaim, which could resolve fairly quickly.

Changes to the DICTIONARY:

Avoid getting the time in a silent GetInfo call. This should be faster.

Passed the new cancellation variable to GetInfo. This is used to break out of the very slow READER process.

Defined the TOKEN flags. The concept of Syllable, and of Paragraph are new. Another reserved token flag to signify selection. Another predefined flag for spelling error, another flag for common words. The remaining 2 bits are flags available to the user for general selection and tagging in the Word Table Editor.

Moved the definition of the primary grammar to the header file and exporting it to users.

New DICTIONARY flags: flSortDescending used in the DICTIONARY::Sort method to sense the cancellation condition, and the new flInSort flag to notify this state to the dictionary editor's status bar.

Explicit definition of the sort criteria enumeration. Added new sort criteria, one for each token flag.

Added the DICTIONARY::SearchString method. It takes two configuration flags to ignore casing and ignore spacing. NormalizeSpaces is a new function that replaces any CR, LF, TAB sequence to just one blank,

GetInfo had to be enhanced to take a limit argument and stop decoding when enough text was uncoiled from the archive. We're using this new feature to get the preamble of each text. The Word Table Editor is using it to populate the word table. The Sort by name logic is better now and interlaces letters, words and such texts preambles. The Sort logic has been corrected and vastly extended with limits, more criteria, and pre-existing orderings.

There is a problem breaking out of GetInfo as soon as the limit is reached. Until I fix it the caller must add another 256 bytes to the info buffer space to prevent an overrun.

The new GetWordName is now returning these limited hints instead of the former <I1234> silly label.

SetInfo also takes the info length argument, but isn't using it yet. It will be used by the future grammar-based searches.

Added DICTIONARY::GetContainers method. It fills an array of sentences where the given word appears and an associated array of counts showing the weight of the given word within each sentence. The function returns the total number of sentences that contain the given word. This method decodes the entire bit-stream looking for a particular word.

DICTIONARY exports the names of the sort criteria and the names of the token subtypes so that the GUI can be more abstract.

Found and fixed a problem with constructing an empty dictionary in advance of running the HTML compiler. The primary grammar had to be initialized.

Deleting the HEAD region of a loaded HTML file.

}

FaxCore {

Updated after a long series of changes to the abstract core and to the base micro kernel.

get_Type is obsolete all throughout the fax core. The new mechanism of type registration imposed by the base Micro Kernel is DNA driven. Any installable core first builds its local DNA vector, then copies it over to the OBJECT_BASE::Dna at the appropriate CLSID place. All type masters are RELATIONs as the new abstract core dictates.

CAT_FaxCore is a vector relation. Please see ANNA K5's SDK documentation for details.

CAT_FaxCore and TFaxType are selectable in the schematic editor, when FaxCore links to OS.DLL.

Fixed a problem in CAT_FaxCore::Free to do with deleting instances other than the core itself which should not uninstall the DLL module.

}

GUI {

Scope View moved the New and Edit actions from the File menu to the Edit menu.

WordTableEditor is now an entirely different and mature beast. Before it was a list of dictionary words with the following features:

  1. Filling the Index, Name, Flags, HitCount, Definition, Crc of each token
  2. Refreshing the word table as the dictionary is still accumulating.
  3. Showing the grammar links of the currently selected word.
  4. Previewing the selected token in a RichEdit control.
  5. Sorting by pressing the column headers.
  6. Singing-out the current token in a blocking loop.

Now, the feature set was extended to the following:

  1. Text is listed as a short (256 characters) extract of the whole. The result is much more readable and valuable, and can be globally sorted.
  2. The word grid fills with the new S, E,a,b, and C independent flags.
  3. Added the bookmark control and logic that remembers list positions as they are double-clicked or drag&dropped.
  4. Multiple selection in yellow of words using both the flags editor and/or the ALT-LClick.
  5. Marking flags with X in clear grid.
  6. Draging and dropping the flag checkboxes to move one column to another.
  7. RClick on the flags to clear the entire column.
  8. Multi threaded sort procedure that may be cancelled by clicking again on the column header. Continuable sorts.
  9. Saving and reloading order files.
  10. Selecting in the grammar list box locates the word in the Word List.
  11. The Search String combo box, the search button and the associated search flags to:
    1. Perform a limited (to 256 characters) search or full search.
    2. Ignore casing
    3. Ignore spacing.
  12. Hard -selection in the word list launches the process of finding a word's container texts. The results are listed in a list box with the following features:
    1. Selecting in the containers list box locates the text entry in the Word List.
    2. The containers list box remembers the source word and locates it in the Word List on RClick.
    3. Hard-selecting in the containers list-box reloads the text preview panel.
  13. The text preview panel remembers the source word that loaded it and can locate it in the word list on RClick.
  14. The text in the preview panel receives highlight commands from other controls as follows:
    1. Bold text matches the content of the SearchString combo. If a match is found, then the preview auto-scrolls there.
    2. In blue underline is matched the word currently selected in the Word List. A global option flag enables the same highlight for letters too.
    3. In green underline is matched the currently selected link word in the grammar window.
  15. The text READER is now threaded and can also be cancelled by pressing the button again.
  16. The new SearchString mechanism is threaded and the core replies by setting the A flags for those text items matched.
  17. Column content may be cleared or moved to another before new searches are performed. The core toggles the flag so the overall meaning of this mechanism is sub-searching text with AND, OR, NOT operators. Search results may gradually be moved to the selection (the S flag).
  18. The Gaius5 button is now operational and does the following:
    1. Locates or creates a node labeled "Selection" under the current dictionary scope.
    2. Creates one document for each selected (S flag) dictionary entry.
    3. Labels each such node with a 100 characters long string taken from the beginning of each information text.
  19. Document entries such created would be stored along with the dictionary and are the junction points to Gaius 5 Annotation System who picks them up from there for further processing.
  20. The Word List can be exported to CSV format, only that Excel cannot load more than 64K entries.

Corrected the FaxCoreTypeLib module. Any non-morphic module that installs an editor library by overriding the OBJECT_BASE::OpenEditor pointer must filter the type of the editing object and return a null editor handle for any foreign type.

If opening the editor isn't working through the core registered OpenEditor function, then no filter caught the object. We go to call the OBJECT_BASE::Edit virtual method which is another mechanism for launching editors.

Profile editor is now storing and re-loading the chart content.

FaxTypeLib: Found and fixed a memory allocation problem with strings editing fields. Core strings must be replaced with other core strings allocated through local_malloc and freed with local_free.

FaxDirView: Changes to the GetFaxMetaServer function to make it past un-named kernel nodes.

}


198 Tuesday, October 04, 2005

CORE {

Work to speed-up the dictionary.

Bit decoders and encoders are now inline for speed.

Ran into an expected alloca problem during a long sort: cannot do alloca inside loops because there is no counterpart to free the stack space, and large loops would overflow the stack. Fixed it by encapsulating the body of the for loop in a function.

Some logic to do with versions of the dictionaries.

Tightened the letter decoding loop in GetInfo. Expanded the Decode logic everywhere for speed.

Corrected that problem with breaking-out of the GetInfo loops when the infolen limit is reached. We don't seem to corrupt the dictionary now. Found that sorting for flags was much slower than sorting for hitcounts. That's because of the shifts we did. Promoted the bit shifts out of the loops. Added profiling to the Sort method so we see its progress in the Provile Monitor.

Removed the IndexEncodingErrors monitor from Decode as those comparisons were quite expensive. Expanded the decode logic inline to the GetContainers function. Added profiling to SearchString.

Added more profile counters to the dictionary: SortOuterLoopIndex/ SortInnerLoopIndex, LastSortEffort, ContainerSearchPeak, ContainerSearchMatches, StringSearchPeak, StringSearchMatches.

Preparing for the sensor of HTML special features. STag now has a flag to say which tag is normal for text and which isn't, and laso has a hit count. A global SpecialHtmlFeaturesFound flag set by ParseHtmlSection.

}

GUI {

The Editor for Dictionaries needs one pass to find the number of containers of a word, then a second pass to actually fill the buffer allocated to that number. It's too slow, so we say the following: the number of texts that include one word cannot possibly be greater than the word's hit-count. If the hit count isn't too large, we allocate the buffer to this size, and get the real containers number from the second pass. If the word is too heavily hit, we have to do two passes.

}


199 Friday, October 07, 2005

BASE: Added another mode to FileOpen to instruct the OS to not cache the file.

CORE {

Found a nasty problem in the bit-shifter bmemcpy routine: forward copy didn't work if the last few bits span byte boundaries. Ignoring the high bite of AX was wrong. Also, with multiple threads and large stacks we get to a point where the distance between a place in the encoding band and some stack parameter that needs loading can switch signs. The bitshifter must do unsigned pointer arithmetic to cope.

It's most unfortunate, but TOKEN had to be exported just so the editor can call GetLink.

DICTIONARY:: GetWordContainers and SearchString now call a response function upon each text matched. Couldn't alocate stacks as large as the containers vector for a word such as "and" that appears 1,500,000 times in the encoded text.

get_EncodingString aligns the return value to sector boundary. The real pointer is stored in TYPE::SuperHandle.

There is a new serializor for version 1 of the dictionary that loads the encoding somewhat faster by not caching the file.

Expnaded the ::Decode method inline and manually optimized its surroundings in a few places, such as ReEncode and GetWordContainers.

Sorting a dictionary's word list by each flag is now much faster (a linear process) speculating that the list falls only in tow classes of equivalence of either 0 or 1. The new sort algorithm throws the items either at the beginning or the end of the list as it traverses.

Added a STREAM flag flCached. Changed some of the code that assumed Flags to be set to 0 after the stream constructor. This is no longer true if the stream is cached and was replaced by a MEMORY_STREAM. Such a stream must set the flCached flag, to inform any of our new loaders that the file is cached and cannot be closed and re-opened for direct mapping.

Files larger than 10M are not cached in the morphic core.

}

GUI {

Found problems closing the local editor when the object gets deleted. Timers may still be active. All timer driven refresh routines must check to see if GetObject returns a non-0 pointer.

Any timer refreshing editor sheet should first verify that the respective page is indeed showing, before doing any expensive loop. This is now done for the refresh of MemoryMap.

Changed the Editor for Dictionaries (EDICT):

1. All particle types are now represented by a bitmap in the list views.

2. The word list now begins with the name field. The index is the last column.

3. Grammar and Definitions panels are now List View controls.

4. The potentially immense list of grammar followers is now built as the user scrolls down the list.

5. The list of definitions is now built on a callback from the dictionary, to eliminate the use of huge buffers of results. The list is now populated as the core progresses for a much better feedback.

6. The search for word containers is now cancelled by hard-selecting in the word list a second time.

7. The string search results are now also logged in the definitions list. Again, this list fills progressively as the search thread finds matches.

8. Some re-arrangement in the toolbar.

9. Added the Filtering system. The Refresh button is now the Filter button, which has a drop-down menu with all the word types supplied by the dictionary core. Initially they are all checked, and the user may uncheck them to hide the respective type from the general word index.

10. Added a count to the word highlight logic of the preview panel. Clicking on a word repeatedly in the list scrolls its occurrence into view.

11. The Sort thread tries to relocate the selected word after the order was changed.

ScopeBrowser: Changed the titles of the Save and Open dialogues to show the node being saved or opened. This is needed for orientation. Turned on the overwrite protection.

}


200 Saturday, October 15, 2005

CORE {

Added a protection to FindCreate to allow it to be called with no separators string. It is often too much for the caller to have to know the layout of separators, when he simply wants a flat search.

Dictionaries can now be created to use either aliased or tight compression.

New dictionary profiles to see separately the number of new HTML, XML, or plain TEXT files imported.

The dictionary's import routine is now checking the actual content of a file to see that it's a HTML. If it isn't it creates just one scope node with the flat test as Information. The profile for TrivialHtmlStreamsIn increments only for nodes that set the SpecialHtmlFeaturesFound during the ParseHtmlSection process. If the ofRejectSpecialHtmlFeatures and SpecialHtmlFeaturesFound flags are set the scope is ignored.

There is a new reserved dePrelinkAlphabet word that wraps a crisscross of letters meant to pre-link all the letters and avoid morphisms.

A new dictionary method RefreshSchematic to construct a schematic of the grammar to help us write the documentation of Edict.

We had to have the tiniest possible dictionary for a demo in Edict, so this version disables PUNKT, NUMBERS and RESERVED_WORDS, and limits the ASCII range to 127. I think I saw some errors in the grammar for the "diction test".

The following HTML markers are said to be common FONT, HTML, META, P, TITLE, HEAD, BODY, BR, B. They can safly be turned to plain text without any loss of content semantics.

GeRELATIONSize is now checking the dictionary's flTightEncoding flag and if set would return exactly the index size. Otherwise the index is aliased through a table. By now you should know what GeRELATIONSize is and does.

A new dictionary always points to the InternalProfile until a Profiler attaches. We had no other way of loading the profile data stored along with the dictionary. The TotalEncodedLength profile is now expressed in bits.

Had to correct the ReEncode method to cope with the old but new flTightEncoding scheme. Morphism of the encoding is now done after all the symbols were shifted. This is to protect the debug messages in getting symbol names.

Added profile counters in the ReEncode to be able to plot the From, To, and Rotor of the morphism process in the histograph. This is now an extremely valuable tool that can truly describe graphically what a morphism is.

The new Encode method is now avoiding to go into the memory manager for relocation when it can determine that InternalFreeSpace is still available for a word.

}

GUI {

EDICT: added another button to generate the schematic of a grammar.

Profile: added another save-as file type to save the chart as enhanced metafile.

Dictionary options: new flags for tight encoding, sort reversed, and reject special HTML features.

}


201 Saturday, November 19, 2005

CORE {

Working on a new and better compression scheme, which currently has one nasty bug.

TestLinks, GetLink, SetLink, GetSize, NameCompare, ResetLink, get_LCount, set_LCount, have all gone from TOKEN and became the responsibility of DICTIONARY. Remember what we said that TOKEN is opaque and only known by DICTIONARY.

New profiling counters: TotalNormality, TotalEType, TotalMrt, TotalNumbers, TotalNumbersSize, TotalEncoding, TotalOther to watch the ratio between pure grammar encoding and exceptions.

No more NUMBERS in this version. They will be encoded as new words.

The text differential also returns negative values.

Changed the bmemcpy routine to make fewer memory references. Added a special case of bit-shifting for ranges under 24 that can be resolved in one load and one store. Short jumps. Left out the "fast" parameter to the function as pushes are expensive.

Disabled the 512byte alignment of the EncodingStream to speed things up a bit.

EncodeInt24, EncodeBit, DecodeBit are now aliases of bmemcpy. DecodeInt24 is a wrapper.

}


202 Sunday, December 11, 2005

CORE {

Struggling with the NEW_ENCODER.

Added the new DICTIONARY::flSuspended flag to put the Primary Learning System to sleep so the machine may be used for other work during the excruciatingly long learning processes. Added a TOKEN::Date field coded as long with 12 bits for year to map a range of 4000 years. The encoding is yy/mm/dd/hh/mm most suitable for sorting.

STREAM is allocating a cache with an extra byte in size where a 0 is pre-encoded. This is so that we can use strstr on the memory cache without overrunning the buffer.

Corrected a problem in NormalizeSpaces. Added a SkipBlanks functions. Added the SearchReplace function that takes a vector of replacement pairs.

Added the ResetAbnormalSchema for HTML.

Worked on a complex filter that reduces two duplicate headers of a text and extracts a most probable date from the text.

Added a fairly complex NormalizeDates function. Added the ReplaceDateWithYear function that reduces a pattern used in dating legal text. Added FindDatePerYear.

One less position in the EMrt.

Changes to the EncodeBit, EncodeInt24, DewcodeBit and DecodeInt24.

Caught and fixed some cases of pointer volatility after calls to Encode which can lead to Malloc.

Found and fixed an overflow problem in GetInfo.

Changes to the stepping rules of the ReEncode and GetWordContainers methods.

Nor using the Index anymore.

This version runs to about 3000 texts and exhibits the "adjacent free blocks" memory problem.

}


203 Thursday, December 22, 2005

CORE {

ReplaceZerosWithBlanks is a new filter to clean-up the HTML stream.

EM, I, PRE,SMALL, STRONG, TRIKE, U are now normal HTML sections.

FindTag had a bug when 0-s were embedded in the string.

NormalizeDates and ReplaceDateWithYear are now protected against overrun.

The end condition of the decoder is now the reaching of the limit and the consumption of all trivial transitions.

If the HTML document has a body, then the body is grafted into the umbrella and the rest of the tree deleted.

Date recognition is now attempted even for a document that doesn't have a bold section.

This version exhibits free-block adjacency errors in the dictionary pool after 10000 files loaded. The compression factor reached there is 3.28.

}


204 Wednesday, January 04, 2006

CORE {

Found and fixed a very subtle memory allocation problem in TYPE::Reclaim where on an extremely rare occasion two adjacent free blocks were created. This was due to assuming rather than calculating an object's header size.

Enhanced the Dictionary logic to keep grammar back-links. Each symbol now links back to the construct that ends with it. The back links of letters form the rimes classes of equivalence. For extensive dictionaries, these back-links limit the number of comparisons required to find duplicates. Found and fixed a signed character comparison against an int which lead to an encode-decode mismatch. Added a protection against a case of HTML file without a body. Some cleanup would be required over this version.

}


205 Wednesday, February 01, 2006

CORE {

A change to the logic that notifies the interlock tree in the Unlock method.

The reason for reordering line endpoints after serialization is the need to skip exports. The reordering and the exports are conflicting problems that cannot be solved together. Added the logic to ignore exports but keep the order intact. One must make sure that the saving cone has no siblings.

ABSTRACT::Import has gained two new parameters to mark the top and depth of recursion through the file system directories. This is the converse as storing an enterprise between two depth levels. See Depth Limited Serialization for details.

DICTIONARY:

Added the SourceId number to TOKEN. This links the token to the corresponding file it came from. The GUI can now open the source file for a selected information dictionary entry.

Found a problem in the preprocessor when reducing duplicate titles and the left-hand-side operator is 0-length.

Speed-up the logic testing duplicates by tightening the loop. With the new scheme based on rimes there is no need to check the flags, the reference word and the duplicate are idempotent.

A faster decoding loop in the ReEncode morpher function as well as for the GetWordContainer method. The new DECODE macro is carefully written with direct regard to the code generator from Borland. It minimizes the memory accesses.

Not keeping any node for rejected HTML files.

Found and fixed a major source of trouble when reducing a duplicate and shifting the rime links back one place. Found and fixed problems in set_LCount.

Sped-up the link searcing loop of Encode by speculating the way Borland generates code.

Avoiding to do MemRealloc when there's InternalFreeSpace to increment.

Found that CRLFs in the HTML file have to be normalized to blanks.

Initializing the locale to the system's default so that isalpha coveres more letters.

Preprocessor is now looking for trivial tables and reduces them to paragraphs. A trivial table has one full column, the rest used for HTML spacing.

Preprocessor is now reducing any numeric <SUP> superscript to ^number.

Resolved a bug to do with reinserting the title if title duplication tested negative.

Several changes to sensing and resetting a non-trivial HTML file.

If the preprocessor can't find a date in the file, it flags the token in error.

A new (old) strategy: one unique numeral. The new word sensing loop isolates numeric sequences as long as 255 (a needless limit) and encodes them behind the numeral word <#>. Revived the PUNKT logic that nicely shows how many sentences are there in the text.

Added the rimes as a sort criteria.

}


206 Friday, February 03, 2006

CORE {

Found and fixed hang condition in SearchReplace.

New dictionary preprocessor filter to decode the A-tags and remove declarative hyperlinks. Logic to build the target file name of a HREF A-tag.

The locales are now set in the Dictionary INIT. Letters such as 'tz' are double byte, and cannot be recognized.

Cleaned-up the NEW_ENCODER statements, experimentation is over.

A few more preprocessor aliases to look for and replace.

Some HTML files have more than one <body> statement. Ignoring all but the first one.

Corrected a problem with deleting SUP-tags and A-tags.

Logic to search the Goal5 style of a date and document number, and insert them in the title.

Logic to detect the Goal5 style of MOF line.

Enhanced GetWordContainer to work for letters as well.

}

GUI: Added a local menu to the dictionary button that opens source files. The actions are to delete the source file associated with the index currently selected, or to delete all non errant source files.


207 Monday, February 06, 2006

CORE {

Fixes to the HTML A-tag parser.

Found another problem in the dictionary decoder: at the end of an info we may have separators past the end of the last word. These were encoded but not decoded. Changed the decoder's end condition.

Found that special words such as Lambda, Punkt, Number, followed the old style definition. Redefined them as the trivial null->null transition that yields a 0-length token. As a result, the initialization of the grammar could now change with Lambda, Punkt, Number to be words instead of letters.

Gave-up the flEnterpriseDictionary flag and its implications. I've got new thoughts about the dictionary as a mountable file system.

New logic to detect recursive HTML tags such as BLOCKQUOTE and reject the file.

New logic to render the TABLE tags as flat text bordered between the edicttablestart and edicttableend keywords.

Fixes to the deletion logic for A-tags, TABLE tags, and SUP tags.

}

GUI {

Added dictionary merging capabilities. This is achieved by loading 2 dictionaries into memory, opening the editor of one, and then dragging the other over it.

}


208 Sunday, February 12, 2006

CORE {

Major changes and improvements to the Dictionary:

The hardest in this revision was to tune the initial grammar.

The grammar is now initialized in the reversed order with the following nested levels:

NullTEXT { NullWORD { NullLETTER,a,b,..z }, NormalNumber, Punkt, ReservedWords }, Encoding1, Encoding2...

This is an important theoretical result, discussed in the Concept Report.

Letters are now offset by NullLetter which isn't 0 anymore.

Resolved the issue of irregular encoding scheme with 0/0. The currently chosen scheme is 0,4,4,4,4,6,6,8,8,10,10,12,13,... which minimizes the morphisms for indexes larger than 4. The scheme allows very young tokens to grow rapidly while they're little, then confines them to more exact spaces. The battle for speed is fought with words with low link counts.

Great adversities had to be overcome in ReEncode, GetInfo, and GetWordContainer.

Changed the notion of EncodingStream. The encoding is now allocated as the Dictionary's Information field so as to benefit from the relocation principle of the Morphic Core. The encoding is initialized with an information signature string ending with 0. The allocated information space is of course much greater, past that 0. The user must never edit the information data of the dictionary.

TYPE::SuperHandle is free for other uses now. This field is not relocatable.

The maximum size of the encoding is taken from a new and editable dictionary static field MaxDictionaryEncodingSize. This is the answer to loading multiple dictionaries of different sizes.

Numerals will never be encoded in the old CARDINALS scheme, so the logic was removed.

Found that the NUMBERS encoded digit string had to be decoded before the grammar transition.

Encode, Decode, ReEncode, do not need flags anymore.

CreateWord doesn't have name and len.

Changed the order of some columns.

Removed the unused "removed" argument of Decode. Reduced Decode by disabling the CHASE_ENCODE_DECODE_ERROR.

A large number of corrections and enhancements to the Primary Learning System of the dictionary.

  1. The most important change is the HTML parser that now creates nodes with right-hand-side oriented labels. The tree CollectInfo has to be called the other way around now. Recursive HTML constructs such as BLOCKQUOTE are now handled.
  2. Found that A-tags have affinity towards the next (older) item instead of the Prev. (We really must reverse the meanning of Next and Prev).
  3. SubstituteXML is now done right at the point of constructing a HTML node, not after collecting the text. This move allows better substitutions of strings that would spawn multiple HTML nodes. Found certain 3 digit HTML&#nnn codes that were handled wrongly.
  4. Some of the "Nr. 123. " lines were wrongly zapped by the RemoveFormFeeds which was ignoring the Nr. key.
  5. Parsing of dotted numbers was often wrong, with cases such as 1.023.004. Dedicated a function to decoding such numbers.
  6. New predicates FindWhenWhich, and GetMOF. These functions may be called multiple times in the reduction of Goal5 type of texts. Goal5 texts are looking very neat now.

Found and fixed a treacherous corruption of the encoding which boiled down to be due to the Serialize being called for canvas drawing (by the Scope Viewer building bitmaps) and temporarily setting the Information field to 0.

Commented out some of the breakpoints.

Added a dictionary flag to disable the high-level filter.

Careful: "Could not access a range of %d bytes from file %s.\r\n" is due to the serializer not being deep enough.

}

GUI Dictionary {

The size of the dictionary's encoding string is now calculated between the limits of the Information block.

Added an editor for the static MaxDictionaryEncodingSize of the dictionary. This is the first behavior of the Apply button.

Added a GUI mode that hides half of the complexity of the WordTableEditor, making it suitable for a plain viewer.

Added the Date editor field (currently unused) and a tri-state checkbox to configure the viewing limits on the time axis.

A more abstract column identification mechanism.

Resolved some issues to do with the limits of the Order vector given a filter setting.

Contention between GUI refresh and Dictionary write operations is now (generally) resolved by skipping the redraw of a busy dictionary and posting a "?" on the screen.

The size of the encoding has to be gauged in bits. Multiplied by 8.

}


209 Friday, February 17, 2006

CORE {

More fixes to the Primary Learning System to reduce more of the goal.

Added the new expression based String Searching mechanism. This is very useful. The search rule can now be bracketed and therefore recursive. Added the new search mode where ZERO matches any number. This is also useful.

Found and fixed some more trouble in GetWordContainer.

Changed the filtering logic as the previous one is too slow. The new logic gathers blocks of sameness and then moves them to close the gap. Found and fixed two memory corruptions in the Primary Learning System one to do with &nbsp; not followed by ';' and rather ending the tag, the other to do with parsing the WhenWhich statements of legal documents.

Found that the <number> word of the dictionary appears 4.5M times in the C49K dictionary. The stack for the loading thread that is subjected to morphisms had to be enhanced to 20M.

}

GUI Dictionary {

A much nicer search bar. It features a results counter, a (yet inactive) gauge for search thread effort, has new flags for ZeroMatchesAnyNumber and Complex grammar-based searches. New date logic. New algorithm to locate the source files. Preparation for XREF analysis.

}


210 Saturday, February 18, 2006

EDICT5 {

Developed the date filtering mechanism. One date picker and 3 meanings as selected from the FROM, TO, and current radio buttons. Logic to enable/disable the date filter. Inverting the date filter. Inverting the flags along a column. Working gauge for the search effort. Better looks for the whole thing. Resizing the bookmark control to match the disposition of the columns of ListView_Words.

}


211 Wednesday, February 22, 2006

EDICT5 {

Added the DICTIONARY::XRef method to compare the current text against all its predecessors and create a forward and a backward link if a match is found. Matching is to take the leading signature of the source file and search with it (as well as with some of its variations) through the target file(s).

Changed the ParseHtmlSection so that any non-closed tag (the likes of BR) sets its node information with left hand side affinity.

GUI work to launch and control the dictionary XRef process. Added a menu option to test the dates of text entries and flag them in error if the month or day is 0.

}


212 Wednesday, February 22, 2006

BASE {

The trouble we've been having with the HTML decoder forced us to define another OBJECT_BASE::flInfoAfterChildren flag. Trees may set this flag to change the affinity of their information field from top to bottom with respect to a TREE's children.

When a HTML tag is parsed and found to have other embedded tags we are faced with 3 scenarios: a) there's a leading text to the first child tag; b) there's no leading text to the first child but there's text after the last child's end-tag and the node's end-tag; c) there's both leading and trailing text. The flInfoAfterChildren flag marks the (b) scenario. The system isn't prepared to handle HTML exhibiting both leading and trailing texts that conflicts with those of children. TREE::CollectInfo now takes this flag into account.

Added GetURLPage and GetURLPageToBuffer to HOST_OS. These two require WinInet.

Tweaked the entry point of BASE.DLL to reset the TLS data on exit. There are problems shutting-down the OTP core when another module (such as the SS) is linked to it.

Promoted the virual BuildInsertMenu from OTP core to BASE. Added the new static CollectInsertMenu to BASE to iterate the system's DNA and collect a vector of types that may be instantiated underneath the current node. Added the new installable CreateInstance to base.

}

EDICT5{

Corrected the TableRowIsTrivial function which wrongly let tables with empty cells escape. Enhanced the NormalizeDates to parse dates in the dd.mm.year format as well. Relaxed the requirement to have a leading MOF so that more types of files can benefit from GOAL5's recognition of the Emitent. More work in ParseHtmlSection to make use of the new OBJECT_BASE::flInfoAfterChildren flag and resolve differently the matter of nesting tags within some text. Added logic to track any ill formed HTML constructs, except the recursive BLOCKQUOTE. Added new dictionary profile vectors HtmlMissingEndTags, and HtmlUnexpectedEndTags. The SPAN tag is now deemed normal.

Double-clicking on a text in the word list doesn't trigger the GetWordContaines logic that would clear the search results.

}

GUI {

The name column of the Profile viewer is now sizeable.

}


213 Sunday, February 26, 2006

EDICT5{

Fixed too much text past the logical end of the file.

Found and fixed an overflow in the relocation of the ENITENT.

Fixed Emitent/document text When Which remaining un-modeled.

Fixed LEGE, HOTARARE, DECIZIE remains capitalized.

Fixed a truncation of the first section of the text.

Fixed titles inadvertently moved to the end.

Fixed HOTARIREA, duplicate titles failing to reduce.

Fixed HTML still present in the encoding.

Fixed the document type which is separated from title by a blank rather than CRLF. {ORDIN | LEGE | DECRET | DECIZIE | HOTARARE | ORDONANTA DE URGENTA} {pentru | privind | cu privire la}.

Less problems, but D49232.BIN still has wrongly coded texts (estimate 20%).

}


214 Wednesday, March 01, 2006

EDICT5{

Dictionary WordTable is now chained back to the dictionary in set_WordTable for relocation purposes.

Added the DICTIONARY::LinkWords(from,to,flags) method.

Split the DICTIONARY class into 3. HTML and SCHOOL are two new classes in the system.

The HTML is an Input/Output interface serializing between STREAM and TREE. It is a category type master that contains by value all Tag types. Tags generate HTML text.

The SCHOOL is the high-level import filter for the dictionary encompassing the roles of the Primary Learning System. School is a memory manager allocating reference blocks. These are formed during the filtering process, and are used to build the legal cross-reference. The XRef now gets created in the following steps:

  1. Allocate a reference block linking to the dictionary text entry.
  2. Scan each text for references of type "legea fondului funciar Nr. 18 din dd month yyyy", and update the reference block.
  3. Loop and cross-compare each previous such reference blocks with the most significant entry of the current block.
  4. Instruct the dictionary to create the links to and from the relating entries.

As a memory allocator, the SCHOOL defines an amorphous relation as its heap.

The School of Law constructs or locates the Romanian Legal Dictionary as well as an instance of the HTML category to use as a compiler.

With these changes, the dictionary works as before. Several of its Options, Flags, and Profiles would need to be removed.

The available memory indicator has now been generalized for all TYPE editors.

}


215 Saturday, March 04, 2006

EDICT5{

There was a great time-penalty incurred in scanning for law types with a string list. Had to finally write a proper parser to Reduce the inbuilt expressions. Added ltCodulFamiliei and ltConstitutie to the law types. Two SReferences compare for strict identity and for fuzzy difference. SetXRefs now uses strict comparisons for law types below ltMOF and fuzzy comparison for those above.

A fix in ReduceDottedNumber. Fixes and improvements to SearchReference.

}


216 Sunday, March 05, 2006

EDICT5{

A series of fixes to SCHOOL::Reduce, SearchReference, BuildReference and SetXRefs.

Added the XRefIterationEffort profile counter.

Enhanced the GUI.

Moved the rimes panel to the center of the browser, peer with the search results list, as I feel they're similar in nature.

Renamed the Followers to be the Derived. Added another list viewer for the Base concepts. Each list also controls a new pair of Edit controls where the focused item gets dumped. The concept is now folding over that of GAIUS. Base and Derived are initially tabs of the Right page control. On RClicking the page control, the Base tab sheet is exchanged with the Left page control, achieving a symmetric view. There's a functional difference between the Base and Derived, in that Derived are the straight dictionary FLinks, whereas the base links need to be collected. Collection of BLinks is done on a separate thread of the GUI. The thread does nothing if the Base tab isn't showing.

}

GAIUS5{

Changes in tune with the growth of the Edict5 concept. The Central editor of GAIUS needs to be split in two portions: top: Scope Browser, and Editors at the bottom. The browser area is one docking station, the editor is another.

}


217 Friday, March 10, 2006

CORE{

Another essential change to the encoder: the Text entry is lead by a raw binary block representing the XRef of that text. SetInfo, GetInfo, DeEncode now take a data block and length as arguments to set and get the XRef data. We have now come to say that a text encodes first as a series of links to other texts found to be referenced, followed by the series of words that make it up. The entire sequence of XRef and words constitutes the definition of the respective text entry in the dictionary. SReference moved from SCHOOL to DICTIONARY, and the meaning of Type was abstracted from the typed enumeration the SCHOOL operates with. This approach has made the school's heap obsolete, with the XRef stored directly onto the encoding stream.

SCHOOL::SetXRef was abstracted from the actual meaning of the XRef type and could now be moved to DICTIONARY. Added another flag to dictionary to instruct it to AutoXRef the texts as it encodes them. We would need to run the XRef as a second pass, so that we first filter the dictionary resulting from the first encoding phase.

The dictionary doesn't need the Date field per TOKEN, as each text now has at least one SReference block associated, which contains the most representative date. This should shrink the size of the encoding by a large number of useless dates for words and letters. Editing the date alone is now done through the dictionary's get/set_Date methods that involve the bit shifter. The profile counters involved in the XRef process have moved from SCHOOL to DICTIONARY.

Some clarification of code in SetInfo.

Resolved some problems in the school's text analyzer. Corrected some states in the Reduce algorithm, to do with LEGEA and HOTARAREA. Added a reduction path for DECRET-LEGE. Obsolete FindDatePerYear. Added the FindAnyDate function to look for date undetermined date references such as "din 12 mai 1997" or "la 15 septembrie 1999". The School's BuildReference method uses this new function as a last resort to auto-date the text entry. In the absence of a main date, the auto-dating process looks for the highest date in text, based on the assumption that a legal document makes references to other legal documents which always precede it. A particular scenario is with annotated legal text where dates could be even higher than the original references. None of these dates can be higher than the release date of the document, hence our assumption should still hold true. Another exception is with texts that embed future dates. The highest of such dates would mark the main date of the document.

Found and fixed a problem with the recursion sensing algorithm of SCHOOL::SearchReference. The performance of the whole Primary Learning System seems to have been greatly impaired.

Found a source of trouble in the HTML decoder when unexpected end-tags were encountered. The scope rules were bypassed resulting in changes in the topics of the input sentences. Unexpected end-tags should be ignored, and the parsing process continued at the same nesting level.

}

EDICT5{

Redefined fl5 to be the AutoXRef enable.

The text dump now underlines the XRef of the given text entry, which is an important result.

Adapted to the changes involved with dictionary dates.

}

GUI{

Added Proxima's logo to the About box.

}


218 Monday, March 13, 2006

CORE{

Added logic that cross-references any text with the null text. The BLinks must become the full list of texts.

Found a problem in the encoding. The normality bit must be coded after the user data block of the null text.

GetInfo is now synthesizing a text by calling the new GetDictionaryInfo. This function returns the dictionary details an should also construct the user details and license data.

}

GUI{

Added EDICT.EXE to the build. This program links in the EditWordTable form, with a slightly different constructor that reduces the GUI to be a viewer only. New logic to rely the panels.

Added the About tile for Edict and Gaius, but they can't be launched from the Editors.

Worked on the Logo of GAIUS, the white balance with balls on its plates.

}


219 Wednesday, March 15, 2006

CORE{

Added XRefSize to the dictionary profile. We need to know how big is it with respect to the rest of the encoding.

Decrementing the previously incremented InfosEncoded profile when catching a duplicate.

The special HTML brvbar character specifier had to also be modeled.

Found another problem with the de-blanking of tags in the HTML parser, upon returning from recursion to continue the loop.

Found a problem in the DottedNumber decoder manifesting similarly in GetMOF. Fixed by incrementing past the dot.

SpecialHtmlFeaturesFound was left floating in the newer versions. Bound it to HTML::ResetAbnormalSchema.

SCHOOL relies on DICTIONARY for options and flags.

Found two parser problems in SearchReference to do with roman numbers.

}


220 Monday, March 27, 2006

BASE{

-------------------- VERY CONFIDENTIAL ------------------------------------

Added SSerialNumber to BASE. Great changes and improvements to LICENSE. LICENSE has PSN (Product Serial Number), UID (User ID), and KEY to match the other two. ProductName and CompanyName are for display purposes only. They get shrouded against the PSN so as to not appear in clear in binaries. SSerialNumber is a 20 bytes long structure, 4 bytes wider than a UUID.

The new scheme is as follows:

KEY is built as a sequential UUID plus a time_t pre pended.

UID is a Windows UUID with a prepended MID - machine identifier. The MID is built by

All serial numbers are kept shrouded with SSerialNumber:::ShroudMask using the BitReorder function so that the user cannot see their fixed or variable parts. UUIDs are constructed with HOST_OS_RT::GetUniqueID

SSerialNumber can be exported to string and imported from string. Match deals with all the details of relating PSN, UID and KEY, as well as calculating the users, the expiry date, and the conditions of a license. Generate constructs a license key.

During matching, a local UserID is created. The local machine ID must match that embedded in the UID, otherwise it's a violation by moving the registration data across another computer with a different MID. Then the KEY==GOAL+product+user must hold. Comparison does not include the MIDs because a KEY uses that part of the encoding for MaxUsers, ExpiryDays, etc. Goal is the inbuilt serial number that everything must boil down to. If the key doesn't match we go to the policy of 1 month courtesy period from the creation date built into the product serial number. If the key matches, the expiry date will be PSN+KEY.ExpiryDays.

OBJECT_BASE::VerifyLicense now passes in and out the UID, KEY, expiry date, users, and conditions. The VerifyLicense method is now used to set, get, as well as to verify the license data. If the license key matches, then a further test is carried-out to see for how long has the program been running and catch any attempt to roll back the computer time. This is done through the new SenseDateRollback(max_time) methods. If the current time is found to be less than the start time a penalty of 10 minutes is applied so that the expiry period moves on. Sensing the rollback involves finding the user profile and collecting the maximum file time of some of its content. The logic here is that the user won't be able to stay rolled back for too long and would have to use other programs at normal time, therefore leaving traces in the TEMP. HOST_OS::GetLicenseKey now works with binary data.

Another matter that we tried to solve is the user limit. We're creating a semaphore with the initial user limit count and we wait on it to decrease it. LockLicense call returns false when the wait times-out or if the rollback is detected.

}

CORE{

Added OBJECT::LoadStore(filename).

Changed the licensing scheme throughout the system.

Added the Archive level to DICTIONARY.

Redefined the Disable dictionary flag. When licensing condition are wrong, the flag gets set and the CODEC refuses to decode any text except ARCHIVE0.

DICTIONARY::sgContinue is a new mode for the encoder where more text can be added past the end of an encoding. When called to continue a sequence, SetInfo doesn't create a new word, and also doesn't close the sequence or looks for duplicates.

GetInfo rturns a special file for entry 0: the signature block of the archive, including licensing and profiling info. That's done by calling the GetDictionaryInfo method.

Defined the flInSchool dictionary flag.

Dictionary::Initialize now takes two more flags: udata and ulen to set the first archive's binary leading sequence. The initialize would really be part of SetInfo. We'll have to consider this for the next cut. Initialize now constructs a grammar with no spaces between words. Added 3 SReference blocks to Text0 for future use.

Added the VerifyLicense method to license dictionaries. The method gets the serial number, product name, company etc from the NullArchive entry, which is encoded. If license is wrong, the disabled flag gets set.

Found and fixed another problem in the HTML parser that lead to texts with truncated start. The HEAD tag could, in some cases have a tail information string. Deleting this heading is the wrong way to go. We're now clearing the HEAD of all its children but leave it in with its info.

}

GUI{

Made a more complex cross for Proxima's LOGO to go into ANNA K5's about box.

Implemented the licensing system. There's a new form on the screens of Gaius and Edict that shows only when Archive0 is selected or whenever the dictionary is found to not be licensed. The form has the following fields:

  1. UID. This is the generated User ID, a number that goes into the registry under the company\product key, and is locked to the machine that generated it.
  2. KEY. This field holds the activation key that Proxima Centauri generates in response to the UID.
  3. MaxUsers. This field shows the maximum number of concurrent users allowed to use the system under the granted license.
  4. Expiry Date. This field informs the user as to when the license expires.
  5. Flags. These check boxes inform the user about any special rights he may have.

Activation is a matter of inserting the correct key into the KEY field and then exiting the editing field. Activation is attempted. If successful, the accessory Users, Date, and Flags are loaded with content. If activation fails, the KEY editor reloads with a failure message. The licensing state is verified on a timer each minute or so. The core needs this pulse to check whether the time has been tampered with.

Improved grid drawing by caching the line while the Index being drawn hasn't changed.

Created the KeyGen program. The following fields are correlated.

  1. Product name, as selected from a drop-down list. Loads the Serial Number box.
  2. Product Serial Number box. Double clicking here generates another number with the current date built-in.
  3. User ID as received from a client that requests an activation KEY . Double-clicking here generates a user id.
  4. Expiry Date, User Count, Fags.
  5. KEY that complements all of the above flags.
  6. A text box to test the shrouding of strings in accordance to the given serial number.

The KeyGen has our logo.

}


221 Wednesday, March 29, 2006

CORE{

Dictionary text references are now variable length structures that allow more than one position to be encoded. Every algorithm that relied upon SReference being a vector and iterated it now had to change to decode from the beginning. The school's SearchReference, AddReference, BuildReference had to adapt. Notable is that a text's reference encoding is now sorted by AddReference using the date, which, type and pos criteria subsequently. This new ordering scheme made it harder to identify the main reference (the text's title), so another field was required for each archive text to tell us the jump location within the reference section where the main lives. It's just like with programs.

Another change to Dictionary is that the NullWord and the NullText are now back-linked to all other words and texts. The consequence is that these symbols make now the segregated index vectors for the respective class of symbols. The index vector for text entries is now used by the dictionary's SetXRef method.

}


222 Saturday, April 01, 2006

CORE{

Added a new method to Compact the dictionary. This function iterates all symbols of the master index table to calculate the minimum bit size of their indexes in the encoding, then promotes a shrinking morphism over the encoding stream to re-encode those symbols under the new size. Had to enhance DICTIONARY::ReEncode to be symmetric and allow it to be called to shrink the encoding stream. Found and fixed an encoding problem in the bit shifter when the source and destination ranges overlap.

Found and fixed a problem in the sort by date path.

Enhanced the cross-reference maker to use the new back list of the NullText symbol. The new logic is that each null element would back-link to all the other symbols of its class, thus segregating the master index vector.

Added the new dictionary GetDateLimits to return the minimum and maximum date references made by the current text.

The SCHOOL now calls DICTIONARY::SetXRefs for each non-duplicate text. Making another effort to date those texts that deviate from the normal dating scheme.

Found and fixed a problem in GetInfo which must be careful when called for verification to not corrupt the udata which might belong to another entry. The null text entry links to and from all other texts.

}

EDICT{

Found and fixed a problem in the sorting method to do with canceling and starting the sort thread.

Added the compact threaded action.

}


223 Friday, April 07, 2006

CORE{

Found and fixed some problems in LICENSE::Verify. Corrected the HOST_OS::InputBox to return the OK or Cancel states. Any expired program would exit when Cancel is pressed on the licensing dialogue.

DICTIONARY::Serialize failed to load of a CD due to a different raw block size and alignment. Fixed by going to 4K alignment. Also, loading must open the file in read only mode. Fixed some problems relating to the creation of an empty dictionary with no serial info. Reduced the allocation of dictionaries.

}

GAIUS {

The main form has a resource with an icon. It overrides the help file of EDICT. It initializes and empty dictionary when the default cannot be found. A stack of 32MB for the compacting thread.

}


224 Monday, April 10, 2006

A very painful exercise: building the BASE and the Morphic Core as a library to achieve the monolithic GAIUS.

ANSI{

Found that strrchr was a copy of strchr. Reverted the logic. Implemented the strncmpi or strnicmp function. Added a prototype for strlwr. Added a LOCALE.H file with a cut-down version of Borland's locale. Corrected a build problem with _ctype table.

}

BASE{

The Base module first builds as the K5BASE.LIB, which then turns it into BASE.DLL with the new dna_base.cpp module that holds the DllEntryPoint.

One really bad piece of trouble linking a BC5 library to a CBuilder monolithic EXE: the CBuilder ANSI library has changed, and several functions such as strstr, strchr, strrch are surrounded by the "std" namespace. I think that this cplusplus-ing of the ANSI library is all wrong. I needed to build my core libraries using my own (very) ANSI headerfiles, but in the end link with Borland's RTL. The "std" namespace was a problem. Had to re-implement strstr, strchr, and strrchr with the "our_" prefix, and change all occurrences with the new names.

The new BASE module doesn't link to WinSQL and excluded any SQL calls with the new NO_SQL option. The FILESYS category would suffer.

BASE_ThreadAttach and BASE_ThreadDetach are new isolations of BASE_LibEntry. This is where TLS is being configured. The new BASE module calls these functions before and after the "try" block of the Fork. This is all to do with monolithic versions that do not attach or detach.

BMemCpy.C moved from ANSI to BASE, so that our ANSI include path doesn't appear in CBuilder. There is no way to override CBuilder's "smarts" in relation to paths and such. BC502 is still a man's tool as far as the settings tree goes. All functions in bmemcpy.c are exported. There is a little trick in base_dna.cpp where a vector of pointers to these kind of functions is built, so that they get extracted from the library and exported out of the DLL.

}

CORE{

Made an attempt to build the morphic core under CBuilder. The problem I struck here is that CBuilder seems to be a single-pass compiler, which means that symbols must be defined before they're used, otherwise the wrong code gets generated. The DNA of the Core is a recursive structure which cannot be initialized in one pass. Had to give-up that thought, TEST.CPP needs to be built with BC5. The project has changed so as to build a few libraries: K5BASE.LIB, MORPH.LIB and MorphTiny.LIB, which is what we needed to build the monolithic GAIUS.EXE. The tiny library is built using the MORPHIC_CORE_TINY option turned on. In this mode, the morphic category is just a wrapper over ABSTRACT adding to it the startup and DNA linkage logic. UIDS.H has to exclude all exports other than ABSTRACT. The tiny library has to be built with private virtual method tables, because their instantiation cannot be controlled at the level of TEST.CPP.

In the TINY model, the SuperKernel has dummy REI and DebugLog RPC interfaces.

In TINY mode the license of BASE and CORE aren't checked. The licensing scheme would be on a per document basis.

The former OS.DLL gets built using the DNA.CPP module which is nothing but the DllEntryPoint to force the Core to export itself.

The BSS doesn't get created if ofNoPersist is set. Needed to take the CD out and be able to close the program.

In this new build environment the operators new and delete had to be cloned at the level of TEST.CPP.

Found a piece of nasty trouble with alloca: a second call in the same scope trashes the 3 stack entries where the registers needing to be preserved are stored. Had to avoid the second alloca in NODE::SerializeFace.

DICTIONARY::Serialize now calls the progress indicator of the stream, if one is attached. We needed that in the splash screen of GAIUS. Found a nasty piece of trouble when initializing the dictionary with a continuation string: SetInfo may end the encoding with a transition to an early symbol in which case the last link to NullWord is wrong. Corrected by getting SetInfo to return the last symbol, and linking that. Also, We want NullText to be linked to itself, and then to all other texts.

}

GUI{

KeyGen is now monolithic.

Added GaiusMonolith to the Navigator project. The splash screen of Gaius/Edict now has a progress bar control, and a bridge class from STREAM to that progress bar. The dictionary serialization updates the bar. Clicking anywhere on the About closes the splash screen. Gauges are extremely troublesome, and had to give-up using one in the About.

EditWordTable now uses more direct ways to build the Order vector using BLinks, when running in reduced mode. Avoiding to do date filtering when the date controls are disabled. Found that rattling the disk to extract the dates was time consuming. Registration form should be visible for any null element, because we now hide the Archive node in Gaius.

Resolved a series of problems with Gaius loading the archive. The file isn't kept open anymore.

}


225 Tuesday, April 11, 2006

GAIUS{

Felt that there's a problem in the license protection to do with sensing the time rollback. Giving the user a 24h leeway.

Added the sgInsensitive flag to GetInfo to be used in conjunction with sgVerify.

There's a new and powerful control to filter the master index with a prefix string. Filtering is driven through the dictionary verification logic which is much faster than getting strings and comparing them. This mechanism seems so poerful that there's little point in sorting the dictionary alphabetically.

Reordered the columns because we're ditching most of them in Gaius, escept for Name, Date, Flags and Index.

The current RLD has a problem: it doesn't start with the null-null link and backlink. Adding these links after loading the dictionary.

Had to enable the MoveLinkTo method to move these new links to the top of the lists where they belong. This is a powerful tool in that Gaius now uses MoveLinkTo(text,0) to move the last opened document to the top of the forward links of NullText. This makes-up for the history of user navigation. Very nice and clean. Little scope for the bookmark control, now, but we still keep it.

Gone back to getting the file name as the label of deNullText.

Added a menu option to select an entire column.

Got rid of the base-list builder thread. Much faster now.

Corrected the Search callback mechanism so that the progress indicator gets updated.

Better resizing logic that makes less noise.

The application doesn't quit while threads are still active. There's a loop after Application->Run waiting for HREAD to terminate all its instances.

}


226 Wednesday, April 12, 2006

CORE{

Resolved the tiny build and the BASE+MORPH library.

The problem with Fork was register allocation by the compiler. We have to have the simplest PUSH EBP stack frame. Corrected.

There was some trouble with the PACKAGE definition in a TINY build leading to the export of the K5Components.

Using the UuidCreate instead of UuidCreateSequential so as to load under WIN98.

WIN98 doesn't allow OVERLAPPED IO. For tiny builds (that we want for GAIUS under 98) we're just doing sequential IO.

Had to insert a GuiPeekDispatch every progress indication so that the About box can get refreshed. This has become a problem because of the non-overlapping IO under WIN98.

}


227 Wednesday, April 19, 2006

BASE{

Tested and corrected the license expiry condition which didn't work.

Fixed the static key recognition logic. Once again, static keys are UID independent.

Enhanced the evaluation period from one to two months from the creation date of a PSN.

Shrouded a few strings that were to do with the licensing scheme and could give hackers ideas.

Added another field to LICENSE to override the expiry date. Not really useful as it turns out.

Reworked the __export system so that GAIUS doesn't publish anything in the monolithic EXE.

}

CORE{

TOKEN flags have better names now: Selected, Error, Result, User, and Hidden.

SReference has a structured vector of positions of the new SReferencePos type, which is used by Gaius to do the hyperlink mapping. The IO field is new in SReference, a 2 bit field to mark that a reference is either input, output or both.

The hidden column is never displayed.

There's a new esExhaustive search flag that tells the StringSearch routine to search thoroughly all manner of the string in the target text and collect the position map. SearchOneString is now a public member of DICTIONARY. This mechanism is nou used by the hyperlink map builder of Gaius. Search now works on and within the limits of an order vector.

Dictionary Sort now has a progress indicator and can now be cancelled.

DICTIONARY now transliterates numbers under below the 10000 range during voice synthesis. The result is much better when it comes to art. numbers, dates etc.

The dictionary loader makes sure that the null text is always first linked to itself so that Gaius keeps it at the top and always visible.

Had problems with register allocation in GetWordContainer.

Some fixes to get/set_Date.

Added the possibility of overriding the creation date by setting the date on the NullArchive node.

GetDictionaryInfo now shows the expiry date as well.

Shrouded some strings to do with licensing to not attract hacking in this region.

}

GAIUS{

The interface has been thoroughly reworked, and is barely recognizable. It has become a thing of great beauty and elegance that I'm finally quite proud of.

Aligned the GUI onto the new scheme of a reusable Frame that holds ListView and DocumentView together. This is the elementary browser capable of opening the selected document. Three of these are instantiated on the form and interlinked. One of the browsers becomes the Center Browser to show what essentially becomes the Master Index and views the Master Document. The other two browsers navigate the base and derived reference lists of the center document.

All the desired mechanisms are now in place:

  1. A unified browser instantiated 3 times and interlinked.
  2. XRef hyperlink mapping in the center viewer.
  3. Colour coding blue for base, red for derived and green for found strings.
  4. Search pattern applies to each loading document.
  5. Spin controls for auto-locating references and search locations.
  6. Pick correlation for loading the linked documents in the lateral viewers.
  7. Prefix filter pre-loaded with categories of NullWord.
  8. The search results are now shown in the master index itself.
  9. Faster and more stable list refresh.
  10. Green and red list items corresponding to the checks of the list items.
  11. Direct click editing of these flags of items.
  12. Date range controls at the far L and R of the frame.
  13. Voice synthesis working again.
  14. Reduced column display.
  15. Dates in the adjacent browsers.
  16. Hourglass indicators for lengthy operations.
  17. Sorting now possible in any list, also cancelable.
  18. No need for the bookmark control, its role is taken by the RMaster index.
  19. Added the directory picker.

Added the new FrameDictionaryBrowser module, split away from EditWordTable.

No ABSTRACT.RC and BASE.RC in GAIUS.EXE.

Could not reduce the library inclusion phenomena with K5Components so had to include UiProxy, CSpin and CGauges explicitly.

}


228 Friday, April 21, 2006

GAIUS{

Mapped the HelpContext IDs and worte a fairly nicely XRefed help file.

Found that the Search process which sets the Result flag also cleared it when auto-locating, consequence of loading the flags in the editor leading to Clicks. Implemented a skipper semaphore.

------

Changed the About screen with a simpler image, and a much clearer overlay for Proxima Centauri. Clicking on this region now launches the explorer.

The Search thread and the redraw message handler of the main thread were racing over the center ListView data builder. The sub-item building in VCL is not quite right and their memory manager sometimes hangs. Found that one has to wait for the sub items to be empty, and musn't Clear their content. The solution to the race is to define our CM_LOCATE message and handler, and have the Serch thread callback POST this message to the queue of Gaius. Gaius then handles the message by refreshing the lists and autolocating. Autolocating on every hit however, was an overkill so we start a 1 second timer instead, which refreshes the lists on expiry. This method essentially compacts the refresh requests to just one. There's also a new option defined: AutoOpenSR which affects the behaviour of the search items dialer in that when checked the target document is not just located in the index, but also opened and mapped. Global searching can now be started also by pressing CTRL-S. Some help IDs were changed, and there's a slightly better hint mapping scheme.

}


229 Wednesday, April 26, 2006

GAIUS{

As of now on GAIUS has its own readme file.

}


230 Friday, April 28, 2006

CORE{

Found and fixed yet another problem with the DECODE macro. We don't have control over Borland's register allocation, so we have to speculate. The problem here was the build difference between MorphTiny and OS.DLL. Resolved.

Added the Target field to SReference to link the documents outright rather than have GAIUS match targets when it runs. Setting the IO field of SReference during the SetXRefs passes. Doing exhaustive XRef matching now, to link every reference to its target. Hasn't been tested yet, a dictionary creation is still in progress.

Set the IMG tag back to "normal". HTML files containing images are going to be accepted, but without any notification as to the missing pictures.

}

EDICT{

DECRET-LEGE is now its own SReference type. Fixed "Ordonanta de urgenta" and "decret-lege".

Another dictionary with another serial number. Added the version to the name.

}

GAIUS{

A better ABOUT tile with more contrast and higher definition.

Found a problem in getting the word containers when running GAIUS-EDICT. Had to trick the optimizer by incrementing/decrementing the local variable.

}


231 Wednesday, May 10, 2006

CORE{

}

GAIUS{

Pushed the archive expiry date forward by one month.

Every time the core notifies the About tile to progress we do a loop of GuiPeekDispatches to refresh the splash screen and uupdate the hints.

Allowing ALT-selection in the side panels by auto-locating in the center index.

The registration dialogue inserts at the bottom of the main GUI so that the search bar is always cohesive to the central document view. There were scenarios where people had the taskbar set on auto-hide and, when it came on due to mouse movement down the bottom of a full-screen Gaius frame, it (the taskbar) obscured the search bar - an annoying effect indeed.

Added the About and Help buttons.

Added a menu-branch to the local menus of the flags editor holding the mode for inter-column operations: OR, AND, XOR, COPY and MOVE. The operations between columns are now comprehensive.

Added another menu option to EDICT that persists an order to the forward and backward links of the null text. This procedure helped correct the date problems of the V3 dictionary.

}


232 Saturday, July 22, 2006

CORE{

Found problems in the Lock/Unlock logic which were introduced a long time ago when the core stopped using semaphores. Single selection was affected as well as the recursive unlocking needed in file serialization.

Gaius Correlator needs strongly ordered binary files. STREE had to go back to an older style of serialization that is recursing on all STREE pointers without attempting to skip past exports. The file system version went up to 12. Clipboard operations must serialize using the export skipping logic, though. There are really two types of binary streams: strong ordered full closures, and export binaries. We'll have to make two types of files.

}


233 Monday, July 31, 2006

A serious amount of change in this period. The reason for this development is the Grid Test procedure that I need for my exam.

ANSI: Needed the putenv and getenv prototypes in stdlib.h.

CORE{

  1. The beginning of the plug-in system. Plugins are external commands of the core. There is no point in modeling each algorithm within the core, so they better be external loadable modules. These modules cannot be EXEs because our RPC isn't ready, but they can be DLLs that do their work inside LibEndry. The problem here is to pass arguments to the DllEntryPoint of the external module. That's done via the environment. OBJECT_BASE::MountLibrary is the neatest way to load such a module, but it expects a morphic type masters library. No matter, it would still transiently link the core process to it and call DllEntryPoint. Extended the MountLibrary method with the new scope argument, representing the node in our browser upon which the module should be doing its work. The pointer is encoded onto the environment as the MountLibraryScope variable. DllEntryPoint on the other hand would need to deconvolve this variable.
  2. Found a mapping problem with OBJECT_BASE whose size was wrongly equaled to that of AMORPHOUS by the CAT_BASE constructor.
  3. SelfTest has new call_type flags which must soon be named. Flag4=8 invokes the SelfTestMemory routine.
  4. Added another integrity test to RELATION to verify that a relation's connector doesn't have children.
  5. Promoted the complex string search routines from DICTIONARY to OBJECT for everyone to use. OBJECT searches through its name and information field. TREE recurses the search down to the children and then up the neighbors, thus flattening the tree and allowing up/down searches.
  6. Moved the serializor's version up to 12. The new version is a cross between the v11 and v9 or so. It is a v11 for a scope's import/export frontier and a direct pointer storer for the scope's interior. We absolutely have to have this to prevent trees from reordering. This serializor is slower because we have to keep asking ourselves if each STREE it is a child of the umbrella, so as to mark the cone surface.
  7. Added x and y parameters to NODE::ResetBounds so as to recursively set the sizes of a cone of nodes to 16x16. Added the appropriate menu option to the SchematicEditor.
  8. Had to launch the SelfTest on a separate thread.

}

ScopeView {

  1. Has a new SearchBar that works as specified for Gaius. The search bar is permanently visible and obsoletes the old search button. The logic is to locate one node at a time within the tree, then use the dialer to go next or previous from there.
  2. Got a new and rather useful behavior: the bookmark button that creates new proxies to the tree. These clones have the same masks as the original proxies, but they are flagged as flProtect so that the original isn't deleted along with the bookmark. The bookmarks are volatile so they clear on the next ScopeView reload.
  3. Finally resolved the problem with applying the proxy flag changes immediately when checking them in the mini editor.
  4. Added another File menu option called External Actions. In the submenu the list of all ".CMD" modules found in the BIN32 directory should be logged. At the moment we only have the DefaultManipulator.CMD module, which is our grid test generator, and test solver.
  5. The new types created by the ScopeView are now aliases of Void, and have 0 size.

}

Correlator {

Changed significantly to run the Law Grid Tests that I need for an exam. There were some extreme issues to be solved:

  1. Enhanced the compiler to parse lettered paragraphs and create nodes underneath an enum type. This extension allows us to compile grid-tests in text format. Conversely, enums are generated with nodes following in normal heading, not bold. Nodes didn't show in the Correlator in Information view, only in CPP mode. The new option CompileEnums drives this new behavior.
  2. Enhanced the XRef generator with two new options that include/exclude the name or information fields of the comparing nodes. This option was useful for cross-referencing test grids that only have names and no info, against doctrine trees.
  3. Added two new Left and Right editors for commenting the links.
  4. Found a RichEdit3 interface that allows us to set the selection's background. We're now using light blue for base XRefs and light red for derived XRefs. It's much better now to not involve italics and underlines which could be used within the text.
  5. With our Law Grid Tests, we want to mark our guess with underline and the correct answer with green background. That would allow the user to see what answers they got wrong. There's new logic to involve the NODE::vaShowBorder and vaFill flags to carry these two states for each enum member. The generator can be instructed to draw the fill state or the underline state from the Correlator's options. Enhanced the RichEdit_Info panel MouseUp logic to set or reset the fill state of a node when ALT is pressed and to set or reset the node's border state when SHIFT is down.
  6. There's a new and highly sophisticated mechanism that recurses relations and lists their supplier contents. A test grid is modeled as a type relating to a number of individual tests. Each individual test is an enum type with nodes. To show the grid, the relations with a RecursionDepth greater than 0 are re-entered as if they were sub-scopes. Another aspect of this recursive mechanism is the recursive building of the IO trees.
  7. Added pick correlation in the concept view that locates the IO wires for the sub-node where the cursor went.
  8. Greatly sped-up the correlation by using the SelLength instead of Text.Length() which seems to be a performance killer.
  9. The new InIoLocate semaphore is used to avoid multiple reloads of the current concept viewer.
  10. Found and corrected handling differences between LINEs and XREFs.
  11. Better workings in the IO tree viewers with faster correlation and less noise. Clicking outside the tree node is not triggering correlation. Clicking again on the selected node discards the IO editor changes and reloads their content. Clicking outside a node commits the changes to that node's endpoint and link justification information.
  12. DragDrop has been enhanced to allow the creation of links and relations involving any sub-node of the central concept.
  13. Corrected a problem with logging the IO trees when the center concept had cabled relations from itself. In this scenario too many ungrouped wires were inadvertently logged.

}

Law Grid Test {

This is now the first external command module. ScopeView was extended to show external commands like such in the File menu. The module does all its work in DllEntryPoint. It is not a morphic module as it does not add types to the morphology. It works on a topology node, manipulates it, then exits. There are two execution paths: if a node called "Test generated on" is not found, then one is created, else the node is evaluated.

Along the generation side of things, the manipulator must parse the given scope looking for public types to recurse them. If the final scope is a union, then it can be picked as endpoint to a new relation of the generating test. Each individual test that gets picked is incremented along its x angle. The manipulator prefers low ranking leafs. The rank of a union is given by the number of throws less the count of wrong answers, which is encoded as the y angle of the leaf. Thus, a leaf test that has always been failed will have a rank of 0 and would always be eligible for picking again.

Test generation is driven over a potentially large arborescence of leaf tests. Each group dictates how many of its tests will be generated. The count is coded in the group's x angle. The inclination of the groups determine a weighted tree. The generator picks new tests in accordance to ranks and to these weights, so as to not repeat the choices.

When called for a test node, the manipulator evaluates the answers. Each wrong answer increments the leaf test's y angle to the limit of its x angle. A test cannot be failed more times than it has been thrown. In the end, an information block of statistics is appended to the test node for the user to read.

}


234 Wednesday, August 02, 2006

CORE: Work on resolving the Promote/Demote/Up/Down logic with the STREE::InsertBefore method. Not quite right yet.

GUI: Tried to resolve the proxy Reorder method. The new PIN button instead of the KEEP.

Added the new MiniJoystick panel which is mouse-hover sensitive.

Enhanced Gaius Correlator with another flag to drive the XRef creation on words only. This is to reduce XRef pollution.


235 Saturday, October 21, 2006

CORE {

Enhanced the SelfTest integrity tester with a check against orphan lines. The new LINE constructor now promotes the parent argument, as it should have.

Added a fix to the OBJECT::IsType method to allow it to work with incomplete types whose relations aren-t yet bound to targets. This scenario is encountered in the constructors of the Morphic Core.

Found and fixed a problem in STREE_BASE::InsertBefore: pivots who are their own parents should not be Removed.

All Information fields of classes loaded from a Rose model had to be bound to their relative objects, for the Morphic Memory Manager's relocation feature to work. The TRoseBaseObjects created by the rose model loader are aliased to CLSID_OBJECT. This is required in the MMM's relocator.

The relation dialer has new logic to locate each member of an array of solid objects, and get its name.

New NODE shape enumerators: shHollow1, shHollow2, shConvexMembrane, shConcaveMembrane, shBand, shRing, of which just shConvexMembrane and shConcaveMembrane are currently implemented. Corrected the convex node shape calculation. Added a 10% frame to convex shapes. Resolved the intersection problems of cable and convex nodes.

Started work on 3D. Enhanced point and Matrix with the appropriate fields. POINT3D enables the new logic.

Corrections to the geometry engine:

HitTestPoly needs to start off with a point guaranteed internal. The former logic of using the center of the bounding rectangle was wrong. Now using the weight center of the first triangle.

GetAngle now returns the angle between two vectors in the 360 degree space. This is now used to measure concavities.

}

GUI {

A large 8M stack for LawView::SpawnCreateInfoXRef. Resolved the alloca problem in FindTypeStr. Resolved a problem with the RTF compiler of Gaius at the end of the text where formatted and clear text lengths do not match. Removing the leading tabs of headings.

Added a counter of instances to the hint of TypeViewer.

Schematic Editor has finally escaped the predicament of having to allocate a dummy instance of the type of the selected object. The SE was re-entering the memory manager intersecting the bitmap image maker. The local editor now constructs a proxy object that consists of just the VMT pointer, so that the IsType method may be called. This is an important step forward.

}


236 Sunday, November 05, 2006

CORE {

Added a new point::HitTestPoly2 which now works based on angle calculations rather than segment intersections. A point is internal to a concave polygon if the number of concavities of the would be extended polygon is less than that of the original.

Added a new function to return an enumeration with all the ordering scenarios of 4 collinear points.

point::GetAngle now returns the floor value of the angle, as needed by the concavity widest angle gathering logic.

Added the point::operator= for better code readability.

Resolved the matter of exact frames to convex/concave node groups in the following steps:

  1. scale each child node by (child->size+8)/child->size.
  2. calculate the composite point list.
  3. scale each child's origin by a factor of (size+8)/size, where size is that of the group's bounding rectangle.

Concavities were resolved in the following steps. There is nothing easy about this.

  1. find one L,R pair of vertices belonging to each polygon, so that LR doesn't intersect either one of the polygons.
  2. If no such pair can be found, then either l is included into l or l into r. While calculating intersections, decide whether l is included into r or r into l. In an inclusion case, calculating the resulting point list is trivial.
  3. In the case of no mutual inclusion, rotate l and r so that the points L and R are first in the point lists.
  4. Concatenate the point lists and close the new degenerate polygon.

We now have a degenerate polygon.

  1. Ensure enough temporary space
  2. Calculate all the intersection points of l and r figures and insert them to the point list.
  3. Reduce null segments.
  4. Collect the envelope polygon by stepping over the edges and keeping only the widest angles. During this stage, some points of the polygon might be reduced.
  5. Inflate the polygon around overlapping points. This step reduces overlapping points and segments.
  6. Reduce sharp concavities that form external angles of less than 90 degrees.
  7. Reduce collinear points.
  8. If any points were reduced during steps 8, 9, and 10, then a new scan for intra-segment intersections must be performed from step 6.

The algorithm is floating point intensive, and would require the caching of the point-lists of each node. NODE needs a new field.

}


237 Sunday, November 12, 2006

CORE {

Fixed a cable binding problem in the core.

The only scope that needn't have a name cause nobody within knows how to get to it is actually the Root node. That node is the most pertinent to bear the machine's name. The root node is now called ANNA K5 and, when serving, it borrows the name of the machine it is hosted under. The bitmap of MORPH is now the angry looking daemon face.

Added the HOST_OS_COMMS::ip_to_str method.

V_Origin and V_Size of canvas need not be overridden.

This is subtle: a node must have two hierarchies. One is the scope, the other is the override. SerializeFace now takes another argument called "target" to add nodes to a relation's endpoint. We can now override a relation's face so that it has a concave membrane. GetPoints uses this second hierarchy.

User login is now creating a relation between HOME and MORPH baring the name of the windows user.

New logic to clip nodes in SerializeFace based on extent of binding rectangles intersected with the roto-translated extent of the canvas stream.

A correction to the point::HitTestPoly method. We are taking pains to find the first non-degenerate triangle of the polygon to take the interior reference point. Then, a point is interior to the polygon if it crosses the perimeter an even number of times.

}

GUI {

Added the USER editor to the type master library.

}


238 Tuesday, November 21, 2006

CORE {

Deep surgery again.

  1. Merged BASE back into ABSTRACT. The reason for this is the need to clone the abstract nucleus.
  2. Changed the type master referencing mechanism to be relative to a given object. Multiple idem-potent nuclei can now coexist in the Core, and yet matters such as the type master resolution, size, ancestry, relation marshalling can still be resolved to the nearest type masters of the equivalence class. The current morphic core features two independent nuclei, with the second - instance of the first. The core type masters no longer have the static ::Type pivot pointer. All inter-type relationships must be relative from now on. The new object->NearestType(reference_type) method climbs up the containment structures of both the object and the reference type, zigzags until it finds an identity of types or at least an equivalence of ClassID, uses the delta of the reference as offset into the type-pool (nucleus) in which the source object lives, an find the nearest type master that is equivalent to the reference. This is a step towards the non-linking and UID-free core. We want to overcome the limitations of the current global DNA vector, and move towards a tree-oriented DNA, perhaps without a vector.
  3. Changed the numbers theory of the system. Int, Char and Bool now owe their size to a series of 5 constants. These constants are nodes contained in the ABSTRACT nucleus, and instances of their respective type masters. GetSize had to change to recurse to the very end-leaf so as to calculate rather than assume the size of these fundamental types. The result is that the clone nucleus may now shift in size and layout when the binary masks of the constants are changed. This is an important result.
  4. There was a problem persisting the constants. The binary mask of the constant (allocated on the heap) was found through the object Handle which is now un-typed, and hence excluded from relocation. The core now uses the ConstantHandle=TypeInfo.Vector to indicate a constant's binary pattern. Also, reduced any fields of SMapFormat that were not being used in the memory virtualization mechanism.
  5. The SchematicEditor, and indeed the core itself couldn't tell constants and selectors apart just by looking at MaxSize. We've introduced the following distinction: constants have an index of -1, which currently are illegal selector index values.
  6. LINE and XREF are OPERATOR instances. Since operator doesn't introduce any new fields to those inherited from RELATION, their type masters could be mapped by CAT_ABSTRACT's vector of auto-relations.
  7. Expedite the morphology queries somewhat by avoiding the memory protection and relocation mechanisms when the pi (Protection Interface) pointer is null, which is the case for all inbound memory traversals. Avoiding the relocation means duplicating certain portions of GetLasRELATION, GetPrev, etc. to de-reference pointers directly.
  8. The OBJECT_BASE::Kernel would now point to the abstract kernel rather than to the top of the morphic core. The issue here is that the core might be wrapped around in various levels of containment that the micro-kernel should not be expected to track. CAT_MORPH routes abstract method calls such as CreateThread to the inner nucleus, for those expecting the old interface.
  9. In the morphic operating system, at the core level at least, the notion of a USER makes no sense. USER renamed to PORTAL. A PORTAL can either be started or connected. Starting a portal is to run its listener in a thread. Since the portal is a USE relation, it points to a scope that would be shared through with a connecting counterpart. The default portal target is the entire morphic core. When starting, the portal takes the default name of the local computer, and becomes a server interface waiting for connections. A portal connecting as a client to a server retrieves the target pointer and sets it as its local own. When in a connected state, the portals own streams and threads. This entire mechanism is currently under review with the upcoming RPC.
  10. Made some changes so that the GUI doesn't link to any static methods of the core at all. The logic affected is LoadStore, DoLoadXML, GetCommonParent and ConvertCoordinates. The only exports the core needs to make are TMatrix and point, and those only so that the Schematic Editor can calculate mouse coordinates in the reference system of the selected node. Other than that, the linkup is done through the OBJECT_BASE::Kernel pointer.
  11. Added another auto-relation to CAT_COMPILE to map over the grammar relation.
  12. Fixed a contention problem in the memory manager by not m-allocating the palette anymore when making the type's bitmap.
  13. Enhanced the OBJECT::Lock and Unlock mechanism with a filtering type parameter to support the new drag&drop cone selection tool of the Schematic Editor.

}

GUI{

Added the relation info field to the list viewer.

Enhanced the cone selection tool with drop capabilities. When a type is dropped over the button, the type master will act as a filter in the cone selection algorithm.

The abstract aspect of the CAT_ABSTRACT type editor doesn't do anything anymore.

}


239 Sunday, November 26, 2006

CORE {

An overhaul to try to upgrade the satellite modules.

  1. The MEMORY category was obsoleted. It never had a role. The ANNA K4 memory manager is stronger in only one aspect: swapping out and in from other MMs and streams. We are not planning on porting it via the MEMORY category.
  2. Integrated the OTP minicore into the kernel. This is the first step towards superseding it with the new REI+PORTAL mechanisms.
  3. Integrated WCamCore as a module of ANNA K5.
  4. Found and fixed a problem at DNA[0] when the OtpDirView logged in and searched for the OtpCore node.
  5. The morphic core now mounts two type master editor libraries: core and OTP.
  6. A change in the NODE::GetPoints method to avoid the errant slide of a concave group. The origin will not be touched.
  7. Preparation for the introduction of the SERVICE class in the abstract core.

}

GUI {

  1. Fixed a crash when loading anything into Gaius Correlator.
  2. Fixed the crash in the type viewer when selecting instances. It was to do with "enums as ints" compiler option. Added detailed descriptions to relations in the type viewer. Info is such as auto-relation, fractal, etc. Added a popup menu that does nothing.
  3. Added the OTP type master editors to the navigator of Anna K5.
  4. Upgraded the OtpDirView to work with the latest K5 core.
  5. Upgraded, and most likely broken the FAX system, all due to the introduction of the OTP core into the K5 kernel.
  6. Added the type library editors of WCamCore.

}

License Key Generator has become a "product". It needs to be licensed before it can be used to generate keys. The key for itself can only be generated when the mater password is typed. The current version allows the licensed user to keep upgrading the license for the key generator, thus defeating the purpose of it all.


240 Monday, November 27, 2006

CORE {

The Core is under heavy pounding. Reorganizing and realignment for the next development phase.

  1. Renaming of some of the core subsystems was long overdue.
    1. The former ABSTRACT.H/CPP was a misnomer as it held the OBJECT and TREE types. Renamed for OBJECT.H/CPP.
    2. THREAD moved from HIERARCH.H/CPP to STREAM.H/CPP with which it has a much stronger affinity.
    3. CAT_ABSTRACT is the container of all core type masters and the core's inclusion gate. It moved out of HIERARCH.H/CPP and into its own module ABSTRACT.H/CPP which formerly held OBJECT and TREE. This distribution onto source files makes better sense.
    4. PORTAL moved out of HIERARCH.H/CPP and into a new module PORTAL.H/CPP. The new SERVICE class also lives here with PORTAL.
    5. TYPE::ecFlags now continues the LINE::ecMax line of change notification identifiers instead of continuing the enumeration along the STREAM line.
  2. Added the SERVICE type master to the ABSTRACT core.
  3. Added TODO information strings to core objects. ANNA's development roadmap should be self-contained.
  4. Added the vaShowInfo and vaCollapsed NODE flags. Added also the shTree shape type. Enhanced the canvas serializor to draw nodes with bitmap, name and info in a node box clipped by the node's bounds. This feature brings K5 closer to the output capabilities of other systems. The new vaCollapsed flag hides the children of a node, thus reducing the drawing complexity at designated places. Moving towards a tree-oriented class display and towards our own tree viewers.
  5. Enhanced the HOST_OS_GUI:: TextDisplay method with the limit point. The new method uses the DrawText windows API instead of TextOut to support the new feature of drawing and clipping object information fields the Schematic Editor.

}

Schematic Editor: Found and fixed a treacherous problem when factorising the editors with multiple selections: the flag editors were not going to tri-state when the flags of two different instances differed. This lead to inadvertent changes on relation flags even when the user worked solely on their nodal visual aspects. Also enabled node flags 11 and 12 which are the new vaShowInfo and vaCollapsed NODE flags.


241 Saturday, December 02, 2006

CORE {

All classes created by the rose compiler must be able to generate nodes. The enums and code fields are node constants of their respective types. This was the source of a corruption in the serialized data.

}


242 Sunday, December 03, 2006

CORE {

Got the HTTP server working much better now and capable of browsing the bones of the system. Decently formatted HTML code with the directory listing. Understanding URLs.

Better for our demonstrations is that the Core does not persist by default. The SaveOnExit flag is cleared after loading the BSS.

Major integration work on OTP. It is morphically mapped in the kernel now. Fully bound and described in inbuilt text all OTP core types and relations. All logic using the OTP's private Dna changed to use the uKernel's. Remapped and simplified all class IDs of the OTP. Upgraded the depending Video Surveillance Station and FaxSys.

Preparation for clipping lines: TREE, NODE, and LINE all have GetPoints and GetVisibility virtual methods.

}

GUI {

ScopeView: aliasing the loader/storer of morphic binary with enterprise binary. Fixed the thread cancellation problem.

}


243 Monday, December 04, 2006

CORE {

If the core crashes then it very much risks to corrupt the persistence on exit. Must prevent that. The CAT_MORPH::ofNoPersist option is turned on right after core init, after loading from persistence. The user must purposely turn it back on to persist on exit.

Found a hang condition in the OTP minicore when loading from persistence, scenario in which the INIT isn't called, so CAT_OtpCore::Type isn't set. The new core doesn't use static pivots anymore, but rather uses relative morphology contours to deal with equivalence classes. The pivot is required by foreign applications such as the Fax Server or the IRIS Surveillance Station which are due for an overhaul. Replacing any reference to this pivot made by the OTP itself. The effect of this problem was the unavailability of type info on for any newly created OTP instances.

Cannot do "new" within the morphic core. Must rethink this, see who's still using "new" and why in the context of the Morphic Memory Manager. Until "new" is repaired the OTP had to change to use CreateInstance, and then configure every aspect of the instance each time. This is also due to the lack of INIT constructors in the OTP. Fond and fixed a destruction problem with the OTP.

Must urgently speed-up the redraw in the SE.

}


244 Tuesday, December 05, 2006

CORE {

The MicroCore of BASE is amorphous unless the Morphic Core installs, at which point it becomes a NODE to parent the morphic kernel. Changed the SuperKernel's constructor to reflect this. The monolithic build of ANNA K5 was still depending on the runtime DLL. The statically linked version crashed. After thorough investigation of the closure of startup and exit conditions, we found/suspected that our new/delete operators were inadvertently called after the core was deleted by the remaining RTL, due to triplication. Also found that ABSTRACT cloned several symbols of the TYPE source file where it originated. Found that TSpinLock was used in both HOST_OS for sockets as well as in TYPE for memory management under the same name. The monolithic build inadvertently reduced them to one. Cleaned it all up.

}

GUI: Waiting for all threads to exit before leaving the monolithic application.


245 Wednesday, December 06, 2006

CORE {

Another fix to relation stepping: the problem was with constants which were indistinguishable from relations. The new logic sais that there should be no reason why a relation cannot be a constant also. A constant "at home" however, child of its type master that is, will not be taken into account by the relations iterator.

}


246 Thursday, December 07, 2006

CORE {

The relation's endpoints are now made available for scope browsing via the new relation tree aspects taFrom and taTo and the overridden GetTreeAspect method.

Added another line of self test for objects to check the integrity of their proxies tree.

A new host_os function BMP2Buffer that serializes a bitmap (handle) to a char buffer. This logic is now required by our HTTP server to build the bitmaps of the type masters of all nodes being browsed, and send them to the client explorer.

Enhancing the OtpServer HandleURL method to include type master icons into the generated web and then respond to HTTP requests to get the images. Having trouble with the closure of RelativePath and FindCreate methods, as well as with the actual transmission of the bitmap.

Found a way to indicate to the Aspect browser that a relation's to and from endpoints are to be navigated towards their parent. The indication is the ^ character in the name of the aspect. We can now expand relations towards their suppliers, which is an important result.

Found a difference in the spin of the RelativePath builder, which is used to get a symbolic relative path between object A and Z, and that of TREE::FindCreate used to conversly locate them. The IDs used in the path to resolve duplication were the other way around. Changed the spin of GetDuplicateID, FindDuplicateID, FindFlat, involved in such calculations.

One important note here: an ID built by traversing a list from its oldest to its most recent element has better chances of locating later, even when new elements are added to the list.

There is a major difference between the two seemingly identical TREE::FindCreate methods with only one of them demangling instance counts encoded on the path. Is this one we want for the HTTP server.

Posting a warning message to the user instead of getting their screen filled with RTF information fields.

Found in the Lanalizer that the HTTP response to getting a bitmap must have 2 CRLFs.

Painful debugging of the BMP2Buffer function. The bitmaps must have the colour depth of the screen device context. The DIB must be allocated on the system's global heap and locked. This is to do with passing the call straight to the video driver, no doubt. Hardwired the value to 32.

}

GUI {

The About tile allows the user to click on Proxima Centauri and jump to the web page.

ScopeView adds several core integrity tests to the global Info menu. These tests are forked.

}


247 Friday, December 08, 2006

CORE {

Caught and fixed another problem manifesting in the petal compiler: constructing enumerations was wrong, and saving the structures to BIN failed. The reason was , again, the wrong path taken by new and calloc through the GUI's memory manager instead of the Core's. The problem stems from the poorly controllable link order. Fixed by: 1. getting the Rose compiler to do explicit malloc instead of new; 2. getting our new operator to explicitly call local_calloc.

The uKernel is a node. The TypeInfo binds to the Core's NODE type master. The Scope hierarchy is now used as insertion point of the morphic core itself as well as all the other otherwise orphan trees as they get promoted to the root of the system.

Fixed the matter of opening a stream by expanding it: STREAM::LoadContent and StoreContent only work when the stream isn't interlocked. Added the Change(ecHandle) notification on streams for read and write, so that the GUI may increment the daemons.

Locking all the primary TFrame_Relation instances onto the MicroKernel as proxies of the system's Root. This is what GUIs are. Adapting the TIsNodeClient::FindExpand method to the new tree conditions, so that it auto-expands trees when nodes get locked.

ScopeView::LogRoot has now changed in that it logs proxies of all its children in the Scope and the Interlock hierarchies. Allowing a node to be promoted out of any core scope and into the scope of the MicroKernel. This set of moves is to allow umbrella nodes to be installed around the Morphic Core.

This version allows umbrella nodes to be installed over the Morphic Core. When loading a system from persistence, the root of the morphic core is found and inserted into the micro kernel. The converse happens at shutdown. The Root of the morphic core is first located as the nearest node underneath the micro kernel. The root gets detached so that the micro kernel's morphology can be deleted prior to persisting or closing the kernel. In the case of core destruction, the morphic core is first removed from any of its parents, so that the process of destroying the morphology type masters can run without recursing over the Root.

There is a subtle matter here: the destruction of an aggregate A-B-C where A is of type C is recursive in terms of lifespan, the solution to the problem is not to sever the link between A and B like we currently do. The proper answer in the morphic OS paradigm is called degeneration. Progressively destroying the type masters leads to the degeneration of all instances, until they all become amorphous. It's the principle of life on our planet, were death, unfortunately means returning complex structures to their elements.

With the new addition of super scopes, a new problem was revealed: the morphology iterator (GetLasRELATION and GetPrev) no longer expected to meet plain types amongst relations. The algorithms are of tragic complexity, so after quite a battle, I found a place and a discrimination criteria using the DNA vector.

Found that the funnelling of the new and delete operators cannot be controlled in any other way than the C++ way. Overloading them in OBJECT class. Added the positional new operator which unfortunately isn't visible from TSuperKernel, so we had to keep our "hate" buffer around.

}


248 Saturday, December 09, 2006

CORE {

It is hard to believe how easy it was to implement that most dreadful caching system to overcome the schematic refresh performance problem. I was very lucky this time to have benefited from some really old rugged logic such as our memory manager, the Change notification channel, etc. The mechanics of the cache is that a NODE may now keep its point lists in the T.Vector field, it has a new vaDirtyShape flag which is set in the NODE::Change method whenever the ecVSize, ecVOrigin, ecAngle, ecShape, ecFlags, and ecTree notifications are promoted towards the root. If a node is concave or convex, hence has many points on its contour, has a T.Vector, and it isn't dirty, then the point list is scanned to find its length and the list is copied onto the caller's buffer. If the node is dirty then the contour gets computed and the vector is created in increments of 16 points. The immediate effect is that a node only ever calculates its shape when something changes it. The more distant effect, and the one we were after for so long, is that LINE doesn't need to intersect the entire endshape, if it can cache the segment number it first finds, until the end node gets dirty. LINE now uses its Pair[].T.Index to idicate the segment of the endpoint node which it intersects. Lines only intersect that segment now to calculate the exact point, which is a more complex data structure that we don't want to cache entirely. Aside from these changes that have potentially brought the screen redraw within a linear domain, we also added the vaMembraneAffine NODE flag to allow certain nodes to break away from the membrane. This was really needed, as a membrane can be kept in place by a few control points, rather than some important nuclei. The whole caching system inflicted 25 new lines of code. I'm quite happy today, as the schematic editor now seems able to deal with a great level of complexity, quasi real-time.

Changed point::LineAndPoly whose return value was never used, to return the segment which it intersects. LINE serializor now uses it in conjunction with the endpoint node cache.

}


249 Tuesday, 12 December 2006

CORE {

Found and fixed a problem with those changes to the parenthood of the Kernel: at initialisation and shutdown the kernel must be inserted and removed into the micro-kernel with AutoBind disabled so that all cables stay.

Allowing type2relation and relation2type cabling.

Needed to restore the functionality of the AutoLoad stream. Streams can again be grafted onto the skeleton of Anna K5 to represent files more persistent than the core. These streams should be viewed as PORTALs to a disk device. The portal is activated when the user expands it. STREAM::LoadContent is invoked to read binary files into enterprise structures. To correctly achieve this, the Interlock mechanism needed to change. STREAM is no longer interlocking the calling thread upon construction/destruction, but rather on Open/Close. We couldn't have a null stream already locked before configuring it. The new interlock is more precise and doesn't let the canvas thread lock a stream locked already by the Expander thread.

LoadContent works as follows:

  1. returns if the stream is dirty to prevent over-reading some newly created structures that weren't yet saved.
  2. returns if the stream is locked. It's the only way to avoid the constant open/close of such a stream whenever selected in the Schematic Editor.
  3. returns if there are children of the stream. This is to avoid re-reading the stream as the user expands it again.

StoreContent works as follows:

  1. returns if the stream is marked read-only to protect the external file.
  2. if the stream doesn't have children then it clears the Dirty flag and returns. This is to allow the possibility to re-read the file by deleting all previously loaded contents.
  3. returns if the stream is clean.
  4. Saves the enterprise structure to file, deletes all contents so as to collapse the stream tree, then marks the stream as clean.

Together, Load and StoreContent achieve the workflow of persisting branches of the morphic tree to disk.

Found that calling Insert and Remove should not notify Change(ecTree) on the calling node but rather on the parent. It's the parent's tree structure that changes.

Found that calling Change in STREAM::Read/WriteBuf was an overkill with low granulation streams. We now post a Change(ecHandle) notification is more than 1KB gets read. The scheme needs to be refined, though.

Enhanced the STREAM::Open function to try two more options to resolve a stream's path: relative to its parent, and relative to the executable's position. We need this with the new AutoLoad to establish directories with relative content.

Defined the ofEnableClipping NODE option. The clipping algorithm for nodes is imperfect and introduces problems with drawing to a bitmap or to an EMF, so needed a way to disable it. Fought quite some battle here to load the Australian railway net-list into the Schematic Editor. Found that the selectors that we have to create must have a shape scheme that complies to NODE's GetPoints expectation, so the shape field cannot be used by the Runtime monitor as thought before.

A concave/convex empty node defaults to square. Changed the caching system to avoid parsing the lists to find their count, rather, ew now store the count in the first location of the cache buffer like DICTIONARY does for its token vector.

Found and fixed a problem in drawing lines with cached endpoint. Any movement operator must clear the line's Pair Indexes and prepare it for redraw. Corrected the problem of calculating the midpoint of lines with cached endpoints. This was source of erratic schematics.

Another potentially major change effected with one line is that CreateInstance produces selectors if no data size is requested, and Constants otherwise. We crashed loading the Railway enterprise binary because nodes were created as constants and the serializor attempted to also load their data patterns from the binary file.

Had to override GetPoints so that non-instance Relations return the default square instead of a large image of their endpoint.

GUI:

ScopeView: Had to add more protections to our TIsNodeClient against collapsing an updating tree. The core is more aggressive now in terms of threads and Change notifications. ExpandFile was not used, the new STREAM::LoadContent and StoreContent are both threaded at the core level.

Schematic Editor: Loading the NODE and LINE Option fields in the SE upon construction. Corrected a series of places where the loading of the mini-editors was a destructive operation resulting in a Dirty node. The matter was with the angle scrollbars.

}


250 Wednesday, 13 December 2006

CORE {

Aggressively pursuing the goal of turning ANNA into a diagram browser server on the internet.

Needed to revive the serializor's logic to render bitmaps. In the past node names could be set to reference external bitmap files which would get loaded into the schematic when their shape was set to shBitmap. The problem here is that the bitmap remains external to Anna's data store. Reviving and testing that logic. Extended the BMP logic to deal with transparent bitmaps, added the new DrawTransparentBitmap function to the HOST_OS module. Changed the name suffix discrimination logic to sense BMP, WMF and JPG. To address the matter of containing the bitmap into Anna's data store the notion of a constant had to change.

Images are now constants of the size of their bitmap image and headers pointed to by the node's handle. This scheme achieves persistence of the bitmap data. Constants and selectors used to have a different shape naming scheme. Re-organised them to at least have common trailing identities so that shBitmap means the same thing whether it applies to a type, a constant or a selector.

Since images are now internal K5 structures, they must travel as formatted pointers rather than as foreign Windows handles. Changed the entire logic that previously carried a bitmap handle to carry our buffer pointer.

A bitmap is first loaded in the NODE serializor upon the first draw if the handle isn't yet set. The images are loaded into our heap as files (not foreign Windows handles), then turned into handles at the funnel level prior to calling the GDI.

Added new HOST functions to deal with various aspects of bitmap handling, such as the size of one. Added the new Jpg2Bmp function to BASE. The function contains inline assembler and needs to stay into a separate file. It currently renders a JPG file from a buffer onto the given device context. It was born in the OTP-Surveillance Station project, was promoted all the way to the HOST funnel due to its tight Windows binding, and reworked for correctness. Was also changed to not mallocate any buffers and rather use the stack, so as to not chop the heap so much on redraws. The new kernel is linking in the JPGD library (which we maintain) to support the new JPEJ nodes in Anna's schematics.

Changed the OTP core driver to track HTTP clients by anonymously logging them in and creating OtpUser nodes for them. The names are built and matched using a combination of IP, Port, and DNS name - which is difficult to get especially on my ex-domain controller server machine. The OTP server now dispatches the handling of the HTTP command to the User node, to achieve session tracking.

}

GUI {

With the reintroduction of Autoload, there is now much more contention over the Windows tree nodes, especially for our Interlock tree. Collapsing such a rapidly updating tree is very dangerous, and my best efforts to protect it for real-time access have so far only edulcorated the danger. The main problem here is that you collapse in one place, but messages get queued and windows does the handle destruction later, in a way that we can't reliably trap. We must get away from Windows trees and do our own real-time safe one.

Updated the OtpDirView in line with the new strategy of inserting Kernels into the Micro Kernel, so as to get the OTP root node correctly.

Linked the Grid-Test generator and evaluator into the monolith.

}


251 Wednesday, 13 December 2006

CORE {

Reviewed the changes made to support images in the schematic editor and found resulting corruption in the BSS persistence, luckily salvageable. The issue here was that images make for constants of the type that generated them and, according to our morphic principles, the largest image would define the size of the generator type. So, when loading a second image in and as constant of the same type, a morphism occurs over the first, thus corrupting it. The solution for now is to say that images are exempted from the type size calculations, and their role is to be purely visual, with no consequence on data flow. Changed the GetSize algorithm accordingly.

Conceived a way of severing the relationship that is established by a bitmap between Anna's file system and the external one, so that the image node can be renamed and moved around without loosing the image. To keep a loaded image and change its name, one first changes the node shape to anything other than "Image". This move turns the picture into a hex constant. As discussed above, the system must not be persisted in this manner because types with constants varying in the size of their binary masks are not yet supported. Once the picture is turned to a different shape (and disappears), the name can be changed to suit any other needs without having the core loading another bitmap from the new path. After renaming, the shape of the node must be turned back to "Image". The final construct has no relationship with the outer world, and has been "appropriated". The OTP nucleus implements a similar feature for moving virtual directories around. To delete the data of an image constant, one renames the node to a nonexistent bitmap. The constant then turns back into a selector. This is an important part of our method.

Changed the NODE serializor to sore the data field of image constants and store just the handle for Selectors. Changed the 2D serializor to put the background behind a picture with Fill enabled, and a frame around and on top.

Added EMF support to the system.

A number of necessary refinements to the new image manipulation: we load BMPs, JPGs and EMFs right into memory on NODE::set_Name, as discussed. The type of the image is now taken from the actual image itself for two reasons: so as to not depend on the node's name string which is now free to change and second, to postpone and burry the type detection logic below the Core and at the HOST_OS funnel level.

}

GUI: Found a condition where the mini-editors of the Schematic Editor were spoiling the Name field, thus provoking bitmap reloads in the Core.


252 Friday, 15 December 2006

CORE {

At this point ANNA K5 has become a server on the internet. It generates a web page that allows a user to navigate the system's core as it runs. This is an important milestone on the roadmap of K5.

The OtpClient::HandleURL is now serving images to the client browser. When a get image HTTP request is de-convolved, the handler looks-up the Core's morphology looking for a type master called "System Images". It then iterates its instances to find an image constant under the given name. The whole prior development was aimed at supplying these images from Anna's file system. One image called "diagram.bmp" has a special regime: it gets generated on the fly from the user's current directory node. Each user session now has its own Current Directory, which is a wire from the user to the target scope. Navigation keeps changing the wire's endpoint.

Needed to allow GIFs to be loaded into the core, so as to serve them later on the web. They are necessary for transparency. Anna.gif is a transparent bitmap that falls slightly over the drawing board.

There was still trouble with these Image constants in that the handle is not relocatable because it's abstract and the morphic memory manager doesn't manage it. The problem applies to all outbound constants. An outbound constant is one whose data mask isn't contained by the header. To remedy the problem, we get the NODE::Active method to relocate the handle of constants before/after persistence operations. Prior to persisting we also destroy the caches, and after loading we set the dirty flags of nodes.

Had enormous trouble building a bitmap into a memory buffer and turning it into a Windows handle for the GDI. There are two new HOST funnels: CreateBitmapBuffer and Buffer2Bmp. These two are now used to generate the schematic over the WEB. The process is to create a bitmap in Anna's managed heap, turn it into a GDI compatible handle, give it to Anna's 2D serializor to render onto, pass the buffer over to the WEB client as the image file they expect.

Started also the Enhanced Metafile Support which would reduce the traffic to its minimum. The fundamental problem here is that the Windows API does not conceive means of rendering the EMF to a memory buffer, only to a file. We need to implement a rather sophisticated piece of logic that constructs a named pipe for the EMF APIs to write to, and have another de-coiling thread to collect the buffer.

Built the topmost Index.htm, which the web client gets first, into the resources of K5. This is to share the development space with FrontPage, tool used to edit the index page. Changed the first page several times. It currently consists of a few inline frames for our Logo, Anna's face, the diagram, status bars, and copyright bar. It references two frames with Tree.htm and Info.htm where the tree and the documentation are echoed.

Resolved the winsock's name resolution performance problem by querying only the IP of the user on each HTTP request received. The user node is built under the IP:Port naming scheme. Upon its creation, the user node is documented with the full DNS name, which is queried only once.

CanvasStream creation/destruction was unnecessarily locking/unlocking an already locked stream, leading to an over-lock condition that the integrity test alerted.

Two new functions of the CanvasStream, RenderEmf, and RenderBitmap to support WEB clients in getting their diagrams built real-time.

}


253 Saturday, 16 December 2006

CORE {

Felt a corruption problem lurking in the system that the integrity tester doesn't catch. I see tiny used blocks of 96 bytes in the memory scan and don't know what they are. Enhanced the integrity test to verify the ownership of node caches. Had to finally add an option to the SelfTest method to repair the errors it finds. Run into the problem of those unnamed flags, which I've forgotten what they were. Added symbolics for those flags through-out the system. Implemented the first repair action in taking ownership of caches. Added another test console command to invoke the repair.

Added some new NODE integrity tests. A test for a constant's ownership of its data block - to verify that our new images don't remain orphan in the heap. A test for the size of a constant compared with the size of its data block which, considering the MMM's internal fragmentation, might differ, but not widely. Added the points cache ownership test and repair. Enhanced the memory scan to take note of orphan used blocks.

There's a problem with wanting to release the caches upon deactivation. The reason we wanted to do that was to purge all unnecessary data out of a morphic data segment on system shutdown. The problem is that during shutdown, the memory managers progressively turns into a file system, as the protection mechanism evolves. So depending on where the node lives in the heap in relation to where its cache lives, it might be too late for releasing. Disabling the logic. Something much more serious has to happen here such as the loading into the system of multiple cores from multiple BSS files.

To aid in diagnosis of orphan memory blocks I need the MMM's reverse pick correlator working. Implementing several new arguments to GetSize format collector. The enhanced logic of GetSize builds a selector buffer with the relation type-masters and indexes that progressively map a given MatchOffset. Enhanced the GUI Memory Map View to return this selector that maps an offset that the user specified by double clicking onto the map. The new MemoryPickCorrelator method is incomplete as it must also descend into the internal heaps of the type masters encountered.

Intense pain in getting the workflow of that WEB portal going. Work on EMF support. My dev environment cannot open any EMFs with anything. It seems that the API creating them also corrupts the images. I'm currently saving the EMF to a file, as the Windows API wants, then red the file right-after. Must change this. Anna should not need a writable disk to run.

Found and fixed a pairing problem between TREE::RelativePath and TREE::FindCreate when no filter type was wanted,

The EMF created by the canvas serializor for the WEB user wasn't scaled correctly. Had to take into account the resolution of the screen both in pixels and in millimeters. The EMF has the nice/nasty feature of bounding the drawing into its minimal rectangle. When displayed in a fixed size box, the EMF is distorted. Had to fake the drawing of two extreme corners on the EMF canvas so as to enforce the screen ratio. The current NODE serializor rotates and scales for some reason that has to be revisited. The EMF needs to be rectilinear so what the NODE serializor is going to do, needs to be anticipated in the Render2Emf function. Going through lengthy pains about that.

Found the proper way to render an EMF and read it from memory, thus not needing a temporary file.

Found that EMFs were rendering all along, only my IE5 on the dev machine doesn't show them, for some reason, so the many hours of tracing were a waste of time.

Changed the generated HTML for our tree to not include the numbers of the list.

}

GUI {

Added the repair action to ScopeView.

Found and cleaned a hole heap of trash in the TypeView in the logic dealing with instances. Prepared TypeView for difference modes. Enabling the instance list as a drop target from Scope browsers. Added the new RethinkCommonType method that factorises the type of all objects that gathered in the instance panel. This new mode of operation is the beginning of our morphic differential. The Schematic Editor currently has the same feature that it uses to factor its micro editors. The Core has a stronger mechanism built in that it uses to compare and symbolically reduce types.

Had to protect the static options of the Core against the Schematic Editor constructing and initializing its flags editors.

TypeView: Fixed the popup menu of the instances list.

}


254 Monday, 18 December 2006

CORE {

Suspecting an integrity problem with the structures belonging to the Object Transaction Protocol, as they return from morphic persistence. Had to write the integrity tests foe the OTP classes. Added some and also the Active methods which might come in use. Reorganised the HandleURL method. Corrections to the server's reporting of image MIME types, the cause of IE5 not working before. IE5 doesn't center the EMF, though. Loading the main HTML page from Core resources and into the information field of each new server. Changed the client code to look for a server page override first, then load the default page from resources. This way we don't need to rebuild K5 to experiment with server-side HTML. Another effect is that each service can have its own page.

}


255 Tuesday, 19 December 2006

CORE {

Traced, found and fixed the problem of not rendering the 16x16 core type bitmaps. Added the HOST_OS::itBitmapHND image type to represent a Windows handle to a bitmap, thus foreign to Anna. The Schematic Editor doesn't currently render GIFs so it now crosses them out.

Needed more integrity tests and repairs. Our BSS is corrupt. NODE::SelfTest musn't attempt to deconvolve constants with handles that lye outside the bounds of the MMM, these pointers could be invalid. The node repair takes ownership of handles within the MMM's bounds. Scaling the tolerance of constant sizes by the GRANULARITY factor. Widely mismatched constants are demoted to selectors and their data masks gets deleted. Node caches are no longer claimed but renounced instead, with turning the shDirtyShape flag on so that the cache will be reconstructed. The former buffer is now either free-floating and subject to removal from the memory scan repair procedure, or it continues to be owned by another node. All caches are now owned by their respective nodes. There is trouble in NODE::Active, still under consideration. Line cached indexes into missing endpoint node caches are repaired by resetting.

TYPE shouldn't really be constants. Reporting the error or repairing it by turning the index to 0. Reporting image caches of THierarcy, in the absence of a GUI attached. The SelfTestMemory finds and deletes any disowned block.

Passing the SMapFormat pointer into SelfTestMemory to help reverse correlation. TYPE and RELATION override the Relate methods.

Added a procedure for creating contained Image nodes using the new iftImage switch. A "System Images" type master is first located or constructor, then the loading image is a constant of the image type. The name gets set and reduced.

}

GUI: Extended TypeViewer with a handle to draw the map with green and red values to correspond to the identical or different data fields.


256 Thursday, 21 December 2006

CORE {

The demo for Mishu two days ago wasn't too bad, but highlighted a half a dozen problems.

  1. The Tiger Server was more of a proof of concept than the real thing. Navigation was terrible, the images of the core didn't correlate.
  2. K5 didn't load its morphic data segment from the read-only BSS file on the CD, forcing us to copy the EXE and the BSS to hard drive.
  3. The refresh of icons didn't work.
  4. The BSS was almost 6M, full of garbage, yet the data was carefully configured so the file is precious.
  5. Image generation in any way other than through the Schematic Editor was wrongly clipped against a region that was offset by the current node's origin, or something.
  6. Rotated images (OTP) could not be normalised in the EMF rendering process.
  7. The memory scanner runs on the GUI thread and hogs everything down.
  8. One shameful problem is that the Schematic Editor didn't allow drag&drop from the browser to instantiate or relate scopes.

Everything had to start with recovering Anna's BSS disk. Enhanced the integrity test to catch and repair more cases of corruption. Found that the OTP mini-core, the Tiger HTTP server, as I seem to be calling it lately, featured no SelfTest channel although it has a crisscross of complex pointers to manage. Implemented. Found that all strings belonging to the OTP were corrupt. Wrote the repair to disown them. It is better to disown fragments, than to attempt to appropriate corrupt ones, or blocks belonging to others.

Found that our mighty NODE::set_Name image constants builder actually creates orphans. Correcting.

With a BSS now purged of rubbish, where nodes have no caches, etc, a crash is exposed in the schematic as soon as one tried to move a membrane affine object. Found that the caches was referenced right after it was deleted. Fixed.

Had another nasty problem with the Morphic Relocator upon NODE activation, where constants (images, embedded files, Int limits, etc. were wrongly adding the pi->Delta instead of subtracting it. Upon shutdown, the pi->Delta holds the address of the Super Kernel, as the relocation factor.

Added the Relate channel with a switch argument to the following classes: (TREE already had one resolving to a move), NODE where two nodes may be wired together, TYPE where an instance of the source may be dropped onto the target, or a relation from destination to source may be created. RELATION should deal with wires and operators, but currently doesn't. The alterations are SHIFT and CTRL while dragging.

Turned around the taFrom and taTo, so that lines (who do not have taScope) default to taTo browsing direction, which is more important.

Added the CAT_ABSTRACT::iftFile file type, in expectation of the incoming File Transfer Protocol.

Fixed the OtpDirView, which has a major advantage over ScopeView with RPC and with its traversal flags. This browser, however only works with OTP objects, and would fail if ScopeView or Tiger browsers insert non-OTP instances in the tree. Repaired the browser so that it may be used in RPC between two ANNA processes. The OTP root directory is located by direct selection now, rather than by searching unreliably through the tree as before.

The OTP type editors were again a source of orphanage in the MMM as it didn't bind the blocks. We could call the MemAlloc with another "owner" argument so that the backlink gets set there, but that doesn't solve matters when going through the ANSI malloc.

Changed RenderEmf completely. The new strategy is to share the code of Node::Serialize so that the caller sets-up the environment using the canvas stream as the proxy root node, and then call the SerializeFace and SerializeChildren. The node is normal on the screen which is a relief. Resolved the clipping problem also. Enhanced the RenderEmf to do pick correlation, and return one of the selected items to the caller after iterating the interlock glove. This new method is much stronger, and perhaps a good candidate to unify NODE::Serialize and the recursion code inside RELATION::SerializeFace/Dial.

The OTP MiniCore was enhanced to deal with HTML exploration of Anna's kernel structures.

Fixed an identity problem with TOtpServer and another with CAT_OtpCore. Also, OtpType masters return a -1 TypeID, so they cannot be mingled with their instances, like in the ABSTRACT core. Added a sketchy Authenticate vector, which gets called for password protected servers to prompt the user with the login dialog.

The HTTP server is now protected for traversal. The users share everything that their server has deemed to be public. This includes: children (I think), inputs and outputs. The administrator can bypass traversal, naturally. Only the administrator can climb up towards the kernel's Root, past the level of the Server node. User maintains its position in the system through the Current Directory wire. Had too much trouble with AutoBind. Disabling it when changing directories. User current directories are free wires across the system.

The HTTP server is now functional, after a full on war, here.

  1. Got the pages to stay volatile. This was one step towards navigating.
  2. Got the traversal logic going, the key to Anna's security system. Security really means wether or not a process can traverse a wire.
  3. Pick correlation on the auto-generated diagram. This makes for the hotspots-from-hell. Click on a piece of a drawing and the browser would load it. Click on blank and go back-up.
  4. The workflow has changed completely. All our URLs are relative to the USER node, not to the server or to the current directory. I arrived to this conclusion after intense pain with all other options. We always navigate relatively. The workhorse here are TREE::RelativePath and its TREE:FindCreate counterpart. Their null element in the set of nodes is the USER.
  5. Got the order of loading right.
  6. Refined the Index.HTM, which now lives as the editable Information field of the server.

One silly, yet amazing outcome is the recursion of HTML over the Server, where the browser recurses in the Information frame, rendering our HTML. This is the first sign that we are about to become a WEB server, should we get navigation and file transfers right.

}


257 Sunday, 31 December 2006

CORE {

The OTP implements a full-on FTP server now. Once the OtpServer created the FTP user, the entire FTP protocol is handled in the 400 lines TOtpUser::HandleFTP method. The FTP commands currently supported are: QUIT, PORT, SYST, NOOP, SITE, LIST, TYPE, XPWD, CWD, RNFR, RNTO, MKD, STOR, SIZE, RETR, DELE, and RMD, which is everything that Explorer used to log the ANNA K5 disk, transfer in and out branches of files, delete and rename objects, etc. The new FTP daemon is a significant step forward in the development of K5 as it marks the beginning of our networking file systems, and demonstrates from a different angle that ANNA K5 has the attributes of an operating system: persistence, exchange, and inner management. Important also is that FTP now allows us to rapidly add to the OS resources that needn't be built-in. Of theoretical value is the unification of constants that lead to the file system whose files are constant nodes, and whose directories are selectors. The integer, bool and char constants, are now files. Our current implementation prevents file overwrites, which perhaps is wrong. We will, of course carry on compatibility analysis to see how we should really respond.

The OTP server automatically selects between 3 protocols now: HTTP, FTP and OTP. A client is HTTP if it sends GET, PUT, POST verbs. A client is FTP if it wants port 21, and here the logic is too restrictive. A client is OTP in any other case. The FTP and OTP access to the core is restricted by login. The HTTP user authentication is still unresolved.

The OTP HTTP and FTP protocols can now do path resolution with and relying on our TREE::RelativePath and FindCreate using the "/:#" path separator set. The forward slash of URLs has the same meaning both internally and externally to the file system client.

Reviewed the matter of constants, their meaning and formation. Went back to handling the constant data masks along the abstract OBJECT::Handle field instead of the TypeInfo.Vector, which is now free for use in caching algorithms.

Caught a node destruction problem with a shBitmap releasing its handle. Switched the order of drawing node children so that the last child draws on top of all the others. Protected the SerializeConstant 32Bit logic against files typically larger than 4 bytes.

Added a forward slash path separator to the beginning of the URLs constructed by the Tiger server, so that the relative hrefs would not turn into file references, as IExplorer would want to do.

Changed RelativePath and FindCreate to extend and to decode duplicate names only for true duplicates. The first item within a scope is now clean of type ID and scope ID, which is a much better look in directory listings.

Added the SetString macro to aid with binding memory blocks to the objects that own them, All OtpStrings must be bound to their respective owners. Changed GetPaths in OTP to trim the last and meaningless path separator, for nicer listings.

Re-established the AUTOLOAD.BIN portal in the morphic category. In light of the new path resolution mechanism of STREAM, the fully qualified path isn't needed anymore. Named and public Drawing Board user node.

Changed the morphic query GetSize algorithm to use MaxSize as a cache. Allocation and release is now much faster, as expected, with implications in loading of structures which is the main application of constructors. Reviewed the size of Int, Bool and Char to not include the NODE size. Reviewed the constants created by the loader of petal files and added size symbolic calculations rather than assumptions when building class relationships.

Our new FTP daemon highlighted a problem in HOST_OS_COMMS::TransferLine where the last null byte of a string must not be sent. FTP gets its notion of EOLN from the CRLF. Those (binary) others might get it from the connection dropout.

}



258 Statistics for the 2006 development year

258.1 EDICT5 was finished, Gaius5 was born

January 2006

{

  1. Several fixes to the Morphic Memory Manager, as discovered during the lengthy, memory intensive analysis passes of EDICT.
  2. Dictionary: grammar back-links (rimes list) to limit searches for duplicates.
  3. Fixed the reordering problem with saved and reloaded binary enterprise files.
  4. Completion of the HTML compiler, as far as to reduce everything to plain text.
  5. Developed dictionary merging capabilities.
  6. The dictionary grammar is now initialized in the reversed order with the following nested levels: NullTEXT { NullWORD { NullLETTER,a,b,..z }, NormalNumber, Punkt, ReservedWords }, Encoding1, Encoding2... This is an important theoretical result, discussed in the Concept Report.
  7. TYPE::SuperHandle is free for other uses now. This field is not relocatable.
  8. Changed the Interlocking system as the last step to free OBJECT::Handle to be used as probe to the outer world.
  9. The Primary Learning System for Romanian legal text.
  10. Developed the date filtering mechanism for Gaius.
  11. The SCHOOL is the high-level import filter for the dictionary encompassing the roles of the Primary Learning System. The School integrates parsers and dictionaries, and features XREF logic.

Monday, March 13, 2006

  1. Added GAIUS.EXE to the global build. The Logo of GAIUS, the white balance with balls on its plates.
  2. Designed and implemented the licensing scheme. Created the KeyGen program.
  3. Compacting dictionaries.
  4. Building the BASE and the Morphic Core as a library to achieve the monolithic GAIUS.
  5. Grammar based text searches for Gaius (and others)

} Monday, April 10, 2006

258.2 Iris Surveillance Station

Iris became fully functional in April with support for Pixord, Dentec and Vivotec IP cameras. Added support for TV tuners. Multiple surveillance clusters for monitoring. Resolved a heap of real time problems.Upgraded the transmission relay.

258.3 Proxima Centauri's WEB site

The site was developed in the April-May period and published in the www.ProximaCentauri.ro domain on 10/05/2006.

An attempt to register the Gaius and Edict trademarks. Revamped our company image with a new logo.

258.4 Gaius Correlator

July 31 2006-12-31

{

A serious amount of change in this period. The reason for this development is the Grid Test procedure that I need for my exam.

  1. The beginning of the plug-in system. Plugins are external commands of the core. There is no point in modeling each algorithm within the core, so they better be external loadable modules. These modules cannot be EXEs because our RPC isn't ready, but they can be DLLs that do their work inside LibEntry. Added another File menu option called External Actions. In the submenu the list of all ".CMD" modules found in the BIN32 directory should be logged. At the moment we only have the DefaultManipulator.CMD module, which is our grid test generator, and test solver.
  2. ScopeView has a new SearchBar that works as specified for Gaius. The logic is to locate one node at a time within the tree, then use the dialer to go next or previous from there.
  3. ScopeView has the bookmark button that creates new proxies to the tree.
  4. Enhanced the compiler to parse lettered paragraphs and create nodes underneath an enum type. This extension allows us to compile grid-tests in text format. Conversely, enums are generated with nodes following in normal heading, not bold. Nodes didn't show in the Correlator in Information view, only in CPP mode. The new option CompileEnums drives this behavior.
  5. Enhanced the XRef generator with two new options that include/exclude the name or information fields of the comparing nodes. This option was useful for cross-referencing test grids that only have names and no info, against doctrine trees.
  6. Added two new Left and Right editors for commenting the links.
  7. Found a RichEdit3 interface that allows us to set the selection's background. We're now using light blue for base XRefs and light red for derived XRefs. It's much better now to not involve italics and underlines which could be used within the text.
  8. There's a new and highly sophisticated mechanism that recurses relations and lists their supplier contents. A test grid is modeled as a type relating to a number of individual tests. Each individual test is an enum type with nodes. To show the grid, the relations with a RecursionDepth greater than 0 are re-entered as if they were sub-scopes. Another aspect of this recursive mechanism is the recursive building of the IO trees.
  9. Added pick correlation in the concept view that locates the IO wires for the sub-node where the cursor went.
  10. Greatly sped-up the correlation by using the SelLength instead of Text.Length() which seems to be a performance killer.
  11. Better workings in the IO tree viewers with faster correlation and less noise. Clicking outside the tree node is not triggering correlation. Clicking again on the selected node discards the IO editor changes and reloads their content. Clicking outside a node commits the changes to that node's endpoint and link justification information.
  12. DragDrop has been enhanced to allow the creation of links and relations involving any sub-node of the central concept.
  13. Corrected a problem with logging the IO trees when the centre concept had cabled relations from itself. In this scenario too many ungrouped wires were inadvertently logged.

} Saturday, October 21, 2006

258.5 ANNA K5 Operating System Core

Saturday, October 21, 2006 {

  1. The relation dialer has new logic to locate each member of an array of solid objects, and get its name.
  2. Convex and Concave membranes
  3. Merged BASE back into ABSTRACT. The reason for this is the need to clone the abstract nucleus.
  4. Changed the type master referencing mechanism to be relative to a given object. Multiple idem-potent nuclei can now coexist in the Core, and yet matters such as the type master resolution, size, ancestry, relation marshalling can still be resolved to the nearest type masters of the equivalence class. The current morphic core features two independent nuclei, with the second - instance of the first. The core type masters no longer have the static ::Type pivot pointer. All inter-type relationships must be relative from now on. The new object->NearestType(reference_type) method climbs up the containment structures of both the object and the reference type, zigzags until it finds an identity of types or at least an equivalence of ClassID, uses the delta of the reference as offset into the type-pool (nucleus) in which the source object lives, an find the nearest type master that is equivalent to the reference. This is a step towards the non-linking and UID-free core.
  5. Changed the numbers theory of the system. Int, Char and Bool now owe their size to a series of 5 constants. These constants are nodes contained in the ABSTRACT nucleus, and instances of their respective type masters. GetSize had to change to recurse to the very end-leaf so as to calculate rather than assume the size of these fundamental types. The result is that the clone nucleus may now shift in size and layout when the patterns of the constants are changed. This is an important result.
  6. The OBJECT_BASE::Kernel would now point to the abstract kernel rather than to the top of the morphic core, a step towards allowing the core to be wrapped in any number of umbrella scopes.
  7. Type filtered drag&drop cone selection tool in the Schematic Editor.
  8. Integrated the OTP minicore into the kernel. This is a step towards superseding it with the new REI+PORTAL mechanisms.
  9. Integrated WCamCore as a module of ANNA K5.
  10. The morphic core now mounts two type master editor libraries: core and OTP.
  11. License Key Generator has become a "product". It needs to be licensed before it can be used to generate keys.
  12. Added the PORTAL and SERVICE type masters to the ABSTRACT core.
  13. Added the vaShowInfo and vaCollapsed NODE flags. Enhanced the canvas serializor to draw nodes with bitmap, name and info in a node box clipped by the node's bounds. This feature brings K5 closer to the output capabilities of other systems. The new vaCollapsed flag hides the children of a node, thus reducing the drawing complexity at designated places. Moving towards a tree-oriented class display and towards our own tree viewers.

} Sunday, December 03, 2006

258.6 The birth of the Tiger Server

Sunday, December 03, 2006

{

  1. Got the HTTP server working much better now and capable of browsing the bones of the system. Decently formatted HTML code with the directory listing. Understanding URLs.
  2. Enhancing the OtpServer HandleURL method to include type master icons into the generated web and then respond to HTTP requests to get the images.
  3. Corrections and refinements to the (hugely important) RelativePath/ FindCreate methods.
  4. Enhanced the Scope View to browse line and relation endpoints.
  5. Opening a stream by expanding it and storing to it by collapsing it when it's dirty.
  6. Locking all the primary TFrame_Relation instances onto the MicroKernel as proxies of the system's Root.
  7. Allowing a node to be promoted out of any core scope and into the scope of the MicroKernel. This set of moves is to allow umbrella nodes to be installed around the Morphic Core. This allows umbrella nodes to be installed over the Morphic Core.
  8. The funnelling of the new and delete operators cannot be controlled in any other way than the C++ way. Overloading them in OBJECT class. Added the positional new operator which unfortunately isn't visible from TSuperKernel.
  9. Implemented the caching system to overcome the schematic refresh performance problem. I was very lucky.
  10. Needed to restore the functionality of the AutoLoad stream. Streams can again be grafted onto the skeleton of Anna K5 to represent files more persistent than the core. These streams should be viewed as PORTALs to a disk device. The portal is activated when the user expands it. STREAM::LoadContent is invoked to read binary files into enterprise structures. To correctly achieve this, the Interlock mechanism needed to change. STREAM is no longer interlocking the calling thread upon construction/destruction, but rather on Open/Close. Together, Load and StoreContent achieve the workflow of persisting branches of the morphic tree to disk.
  11. The supported image formats: BMP, WMF, EMF, JPG, GIF.
  12. Friday, 15 December 2006: at this point ANNA K5 has become a server on the internet. It generates a web page that allows a user to navigate the system's core as it runs.
  13. The OTP implements a full-on FTP server now. The FTP commands currently supported are: QUIT, PORT, SYST, NOOP, SITE, LIST, TYPE, XPWD, CWD, RNFR, RNTO, MKD, STOR, SIZE, RETR, DELE, and RMD, which is everything that Explorer used to log the ANNA K5 disk, transfer in and out branches of files, delete and rename objects, etc.

The Tiger Server with its WWW and FTP daemon is a significant step forward in the development of K5 as it marks the beginning of our networking file systems, and demonstrates from a different angle that ANNA K5 has all the attributes of an operating system: persistence, exchange, presentation and inner management. Important also is that FTP now allows us to rapidly add to the OS resources that needn't be built-in. Of theoretical value is the unification of constants that lead to the file system whose files are constant nodes, and whose directories are selectors.

} Sunday, 31 December 2006


259 Tuesday, 2 January 2007

Another year, and I'm still here ...

CORE {

Had to convert the BSS to adapt to the change to constants.

Some nodes are nameless and the FTP client can't distinguish them. Enhanced the RelativePath/FindCreate pair of methods with another path separator (the fourth) giving them the character to use to mark nameless nodes. Tiger is currently using tilde.

Added some tests and repairs to the OTP. Allowing the autobind in WWW server, as that wasn't the problem. Enhanced the FTP server to allow file upload with overwrite.

Added caching to the Interlock to speed-up the loading of Railway Demo.BIN. Generalising the caching system at STREE level to anybody calling FindByFlatID. The cache vector doubles in size when an index overflow is detected. For the interlock, the Unlock invalidates the cache.

Sampling some lengthy load processes showed that most time was spent around the GuiPeekDispatch of our overlapped file IO. Disabling, as I've forgotten its reasons.

Caught a crash condition with Type Viewer loading lines.

Found trouble in LineAndPoly function. The return value was always wrong, impacting on the LINE::Serialize method finding the endpoint.

Traced and corrected the throw of node coordinates towards infinity when picked from a wire oriented schematic. Determined that whenever a node discards of its points cache, the caches of all the lines to and from that node must also be cleared.

Protected set_To and set_From against unnecessary reloads, as prompted to by the WWW and FTP servers. Found the source of that haywire wire to be the fact that all OTP objects were deemed to be selectors. Added the IsSelector method for OtpBase to always return false.

Felt that GetCommonParent should, after all, be limited at the top by the Micro Kernel.

Had to rescue the BSS again, from an orphan canvas stream left over. Added the STREAM::SelfTest method to look, report and fix non-null handles. Enhanced and corrected the LINE SelfTest that checks the integrity of endpoint cache information. We only cache end-point nodes with shConvexMembrane or shConcaveMembrane. The other basic primitives get created quickly and line endpoints can assume indexes into their perimeter. If a line is found to index a lost node cache, then the repair clears the index.

}


260 Wednesday, 24 January 2007

CORE {

Found and fixed another flaw in FindCreate to do with locating duplicate IDs. This was another case of asymmetry with RelativePath in that the duplicate name count number must be incremented regardless of their type.

Corrected the core's binary file import logic to create node constants.

Fixes and corrections to the FTP server login logic. Enhanced the error messages.

The FTP and HTTP clients are now proxies of their current directories. No more destructive lines between client and the topology which it traverses. Since OtpUser can now be a proxy to itself it must not recurse through the notification methods. Change is strapped for TOtpUser.

The HTTP server has changed in that it locates its Index.htm rather than use the info field of the WWW server.

Another revision of the HTTP server. Morphology Server (TIGER) is up the version 2.

An entirely new line of thinking: the Morphology Server is a HTTP version of Gaius Correlator. TOtpUser::LogIO is new and similar to Gaius. HandleURL has changed drastically. It locates files in the internal file system and transparently passes them onto the client. The only intercepted HTML page is the URL which is now seen as a meta socket between client and server, one through which function calls are posted to the core along with their arguments. The responsibility of configuring the calls is almost entirely up to the client.

RenderEmf of CanvasStream has been enhanced to draw schematics from the perspective of lines.

RenderEmf was changed to take into account the TLS value when drawing from the perspective of a wire.

}

GUI: ScopeView slightly extended to start searching in fewer moves, changed the file open filters so that image and any other ordinary file would be imported as binary constants of NODE in the inner file system. Changes to the OtpTypeLibrary to allow the core editors to also open. Added a viewer/editor to the OBJECT type editor. This new tab allows the user to view and edit a file constant.

WEB {

This seems to have become a new subsystem of the morphic system.

Editor.htc changed in that loading an object into the editor only ever reloads the name, icon and center info panels, leaving the side preview panels as they where. The editor only loads documentation at the first depth level. Corrected the zoom logic in the side panels. Title.htc needed HTML and head tags.

Tree component has to change the ActivePanel field of the parent control on each mouse click, not just on dblclick.

Panel component now exports its LoadPanel function which should be called from the navigator whenever the IO is refocused so that the center diagram may be refreshed with correlation. The W, S and P codes stand for write, select and pick, and they need to be passed to the core correctly. An attempt to pick correlate the diagram.

A correction to the url navigation logic to funnel it through the GoTo method. This resolves the design time navigation problem. Enabled the soft loading of the Infra editor whenever the IO trees are clicked.

}



261 Friday, 7 September 2007

Resuming after building the DRS Core for RomSoft.

BASE{

Found and fixed a crash condition manifesting itself on XP with HOST_OS::InputBox assuming that the buffer in which the user string would be read is 128 bytes long. The licensing key editing logic passed in a 100 bytes long buffer resulting in a difficult bug. Enhanced HOST_OS::InputBox to promote the size argument all the way into the dialogue loop.

The listeler of the PORTAL hung when a client connected to it. It is a deadlock situation involving the debug log which cannot be handled while the GUI thread is blocked in the listener. Re-enabled GuiPeekDispatch-ing inside the HOST_OS_RT::FileAccess loop on overlapped IO.

Added OBJECT_BASE::IsEditor returning false with the exception of the TNotifyRelation descendents, which are all GUIs.

OBJECT_BASE::GetStates is now checking a node’s proxies to see wether any of them is an editor. If so, the Navigator underlines the node label considering it to be gsLocked as for any interlocked objects.

}

CORE {

Resumed work on PORTAL and RPC.

A remote object is one which is ultimately contained by a PORTAL. Enhanced OBJECT::GetStates to sense this condition and return gsRemote. Portals override the GetStates method and return gsRemote if connected. The Navigator shows gsRemote nodes in blue.

PORTAL can be used as either a server to listen and dispatch or as a client to connect and pause. The possible states of the portal were reworked to reflect these mutually exclusive hypothesis.

Changed the login/logout protocol. It now allows reconnection to a persistent session. Each client command must identify the client gate of the portal. When no server session is found to correspond to the client, then a disconnected session is tried based on the user name or IP of the client. Other changes to the login cross-over logic.

Enhanced the morphic core with command line arguments to specify the BSS to load.

More thought was given to the notion of HOME, USER and USER_GROUP. The current line of thinking is that the HOME is a memory manager for all users of the Morphic Core. It doesn’t have to have its own heap. The HOME is a containment relation of MORPH. HOME generates USERs, which we don’t have in the system as such. We therefore re-assign type master USER_GROUP to mean USER. A user is an instance of HOME. As a type master, USER too generates objects that are isomorphic to PORTAL. So users generate portals.

}

GUI {

The Portal alters the usual block containment by claiming all proxies to remote objects for itself. The current portals, however, don’t manage their own heap an rely upon the global Memory Manager for allocation. Memory block ownership by a portal is a logical grouping used in relating proxy objects to their remote targets. The Memory Map Viewer needed a protection against memory managers with zero-size heaps such as the Portal.

Changed the Portal editor's login/logout and start/stop logic.

}

Morphology Server {

Chased a startup defect of the Tiger interface to the morphology server. No initial context was established due to the first HiddenPage.HTM coming without any Name, Img, TypePath and NamePath fields. Protecting Defult.LoadAll against such conditions. The mouse tracking logic in Panel.HTC needed protections against NaNs, needed to be brought back to 0 when in non-diagram modes, and protecting against mouse movement over a failed panel with no width or height, all of which were raising exceptions and disturbed the rather delicate flow of Tiger.

}


262 Wednesday, 3 October 2007

BASE{

WEB users are proxies to the node of their current directory. As such, a race develops in the GUI notiffications for the ScopeBrowser. Needed to use (again) the skipper semaphore in UPDATE_PROXIES. Some stream handles such as the socket cannot be accessed at random positions. Seek would return -1. A protection in HOST_OS_RT::FileAccess against this case.

}

CORE{

FindCreate was enhanced so that objects of types descending from that which was specified may also be matched. This change was reqired by the Morphology Server which is now based on portals instead of OTP. Exact type matches now require the FIND_EXACT_TYPE flag to be set.

Enhanced the file open logic to create constant nodes for each file of a WEB site by treating the itNone image type as pure binary.

Enhanced the editor for PORTAL so that it persists at least the Parallel flag, which is presently needed by the FTP dispatcher, along with a few others yet unused.

The instances of the USER_GROUP now have relation fields and are born as relation instances. This is to expedite the creation of portals in the user group space. GetPaths, LogIO, BuildInfoTree, ConvertToHtml were promoted from OTP.CPP to PORTAL.CPP. TOtpUser does’t need the cmd to be passed in. Added the HandleFTP, HandleFTP and Authenticate to PORTAL. In essence, this version has promoted the FTP and HTTP servers from the OTP miicore to the abstract Kernel level. The servers were corrected and enhanced so that we may load Proxima Centauri’s WEB site into ANNA K5’s disk and run it from there. Changes to the Morphology Server to adapt to its new position.

}


263 Wednesday, 10 October 2007

GUI{

Found the source of system performance degradation while ANNA has the GUI opened: have to call Application::HandleMessage so that the GUI thread eventually goes to idle mode.

}


264 Tuesday, 6 November 2007

CORE {

Added the OBJECT::ofChangeNotifications set by default to load structures faster without updating any daemons. The notifications during serialization are a tremendous expense of time. Corrected the OBJECT::GeStates method so that it returns a remote state for an object whose container is remote. A protection to path_compare. A faster version of TREE::GetCommonParent. A faster version of TREE::GetCommonParent. Enhanced NODE::GetName to return the name of the server of nameless proxies. Changes to AutoBind.

Significant work on PORTALS.

}


265 Tuesday, January 27, 2009

Back to work!

Merging the Document Generation and Recognition System into ANNA K5.

Dropped the source pool of DGRS into the build tree of ANNA.

The new directories of C++ DotNet sources are:

Bin

CasaDeMarcat

Distribution

Documents.DRS

DRS Control Library

DRS Control Library CS

DRS Control Library VB

DRS_Main

DrsCore

NXP

SearchFilesOCX VB

Type Editors

UsbLibrary

New into the pool are the C++ sources of

TESS

LIBTIFF

Modified TESS to build as a BC502 static library. Inserted a few checks in some places where the BC version of TESS crashed. TESS now expects the config files into the Configs directory which is peer with the EXE.

Created ALL_OCR.IDE to build TESS as a library along with ANSI.LIB and an OCR tester.

OCR.EXE does the following:

  1. creates a memory bitmap file which it fills with TIF data using LIBTIFF
  2. converts the memory bitmap into a device dependent bitmap handle
  3. invokes the DTK as a borland import library to detect any barcodes on the page
  4. Initializes and runs the linked in TESS library to recognize any text
  5. Has problems closing and cleaning TESS-up.

This is the foundation of a new imaging driver.

ANSI {

Needed some extensions to our ANSI library for locale and character sets.

Corrections to the prototypes of fread, fwrite and fopen.

Added ptrdiff_t to stddef.h

Added qsort and abort to stdlib.h

The new _endthread function that TESS uses.

TESS uses setjump. We extended our ANSI to cover those structures.

New prototypes for vfscanf and rewind which TESS uses.

Had to change the declarations of our signal function so that we compile with our ANSI header files but link to Borland’s runtime library.

}


266 Tuesday, March 24, 2009

Had to put a protection in the file IO so that HOST_OS_RT::FileAcces doesn't call the operating system for 0-length read and writes. The issue arises with a nameless value in the latest MORPH.MDL file.

Needed another protection in the FileAccess method to bail out at the end of the file where GetOverlappedResult should be tested for errors. This seems to be a slightly new behaviour of Windows 7.

The issue with the file IO is that overlapped read is allowed under Windows7 64Bit past the end of a file which was opened for read. We had to test for filesize in our stream de-blanker to avoid errors. There's also a new behaviour with overlapped IO with the WriteFile API occasionally returning an error suggesting that the file isn't ready.

The OBJECT destructor does a hidden reset of the VMT just before the opening brace. We have to reload the correct VMT from the type master so that the correct DONE virtual method may be called. The issue became apparent with the Rose model loader which creates a STREAM on the stack but could not destroy it to close the underlying file.

Some corrections to Gaius merging hierarchies.

FIND_EQUIVALENT_TYPE is a new mode for OBJECT::FindCreate. Consolidating type filtering under the new IsSameType function.

We are now allowing the creation of 0-sized memory blocks in the morphic allocator.

Renamed the rose type masters to match the model.

The petal loader now creates 0-sized types with node constants. Found and fixed an error with boolean values loaded from the model.


267 Thursday, March 26, 2009

Work on the Rose model loader to produce an object map with correctly ordered fields and without gaps.

The type master viewer needs to know the size of all scalar fields before calling back the plotter. The matter was obscured for the objects of the core itself, since Int, Bool and Char are treated in advance as exceptions to the calculus of size. The rose model loader creates types with the generic user class ID so Int, Char, Bool need to be sized using the MaxSize assumptions.

There was yet another problem with the rose loader as it implements structural morphism when adding relations to classes. The GetPrev algorithm in the RELATION::Grow method was iterating and offsetting relations multiple times. We now filter out any relations that does not induce size and only offset those who do.

Reversed the order of iterating the lists in the class binder of the Rose model loader.

Caught several problems to do with size and granularity of types and relations.

The constants created for Code and for Local Variables are of size so as to not change the overall size of a method's prototype.

We are loading method attributes after loading its parameters so that locals and code follow the parameters.

Tried to define those Int, Char, Bool scalar types that we load off a petal by constructing constants for them, which is the correct way to size a type in advance of knowing its fields. Unfortunately, the GetSize morphology parser needs to know the size of a type in advance for performance reasons.

There were some confusions to do with boolean fields of the petal files leading to either integer or true/false values which could not be then consistently deleted.


268 Friday, March 27, 2009

The Rose category could not be purged due to a type ambiguity surrounding value fields which may be int, bool or string. Value fields and true/false fields are now treated as text owned by the corresponding type master.

Constants such as enumerations are now owned by the type master that introduces them into the scope.

Cleaned-up the model files to yield fewer differences with ANNA K5 core.

Found a way to speed-up the store serialization of enterprise structures by involving the Interlock.Index field to tell the order of a node that has already been stored. GetFlatID isn't used any more. There is still a large performance penalty when storing compared to loading which needs to be investigated further. When storing the serializor calculates the frontier which the loader doesn't do, whereas the loader allocates memory which the storer doesn't do.

Resolved the matter of serializing the new (and exact) structures constructed by the rose compiler. Constants differ from selectors in that they have a degenerate TypeInfo.Index of -1. There are complex reasons for serializor to not store the TypeInfo molecule and rather keep a summary. One reason is our cut&paste logic which needs to resolve gaps at the destination and also restore missing endpoints. I've now introduced a new NODE flag flConstant which is hidden from the user for now, and is set and reset during serialization as a persistent alias of TypeInfo.Index.

Editing the model of ANNA and merging types to match the layout of the core we ran into the circular inheritance problem or fractal containment. The size of such a structure that includes itself is multiplied by the relation's recursion depth counter. For now we only understand first degree fractals. There were (and still are) problems shrinking a type whose fractal containment relation has been deleted.


269 Tuesday, 31 March 2009

MergeNodes eventually tries to locate a sub-node of the source within the subnodes of the destination. It did so without any filter and confused types and lines. MergeNodes only matches TYPE descendents now. Any other type of nodes will be deleted in the end along with the destination when the merge is over.


270 Friday, 10 April 2009

CORE {

The core now accepts the /I /O /U command line arguments to specify the BSS file to persist from and one or more BIN files to load into the user directory. This automation is long overdue.

Enhanced GetPrev with the endpoint direction flag so it may iterate client or supplier relations of a given type. Development of TYPE::GetNext method, the opposite of GetPrev. It had to be passed the endpoint direction as well. It would have to be enhanced with the protection mechanism to work on files.

NODE is either a selector or a constant. Constant may be embedded or external. The difference is that the handle of an embedded constant points right after the node header. Embedded constants are variable tail nodes of size sizeof(NODE)+MaxSize. Any memory walking routine had to adapt.

Serializing embedded constants posed a difficult problem: the CreateInstance that the serializer does for unbound objects had to take the extra size of the constant as an argument. We therefore had to persist the MaxSize field before loading/storing the constant. We struggle hard to not upset the binary file structure so MaxSize is only stored for NODE types and not for descendents.

The Rose compiler creates embedded constants now for enums and for entry and exit code. The constants are first created by the NODE type master and then morphed to the enum class which is compiling. This morphism operation has to preserve the abstract handle, since it points right past the NODE header to denote an embedded constant. SetTypeMaster must preserve the negative type info index that differentiates a constant from a selector.

A new type master reductor algorithm which spirals down the types to distribute outwards USES relations and reduce the reverse xref of basic types. The heavily used types are called too often in size and layout calculations and iterating the rxref is a nonlinear performance penalty. USES relations also help a scope in being (almost) independent from the core. The only interface a peripheral module has with the core is a set of unique USES relations.

Added a spinlock around the calculation of export/import relative paths to get closer to parallel file IO.

Corrected the NODE::SelfTest logic with respect to embedded constants. Corrected the tester to not report bounds errors for invisible relation fields.

There were many problems with structural morphism as unveilled recently while editing a binary model in order to map it over the core. TYPE and RELATION::Grow were corrected in several places. Growth can now be negative as a result of deletions. Grow correctly shifts the offsets of any relation following the morphism point. This algorithm carefully avoids endless loops, static and USES relations, and allows for fractals of a specified depth. Fixes to RELATION::set_Cardinality. A debug tool was implemented to log the reason for morphisms. MorphismPath calculation is currently disabled.

Work on a new kind of a USES relation which reduces the wiring around its supplier type. USE does not inflict size into the client's map. When creating, a USES relation claims all wires going to its target thus becoming their new supplier. LINE::GetOuterCable was modified to return the best cable at the end of which to create the USES relation that would claim all wires going in the direction of the cable. USES does not claim any wires which are tagged as read only. The morphic core is designed with certain convoluted auto-containment relations that must not be so bluntly re-linked. Deleting a USES relation rebinds its clients to its supplier restoring the original status before the USES was inserted.

Re-worked and finalised the type merging algorithm which now loads a full definition binary xref, with classes and methods, right over the running operating system without crashing it. The logic inproperly is assigned to Gaius Correlator and must be promoted to the Morphic Core along with the C++ code generator.

The fully described ANNA K5 core is an important result and a milestone of this research. An OS with full symbolics iterates its maps painfully slowly, and is also slow to persist. The fundamental types of the KERNEL space must be cloned to its clients via USES relations to ease the mapping effort. The road should now be opened to reactivate the instructional compiler in the COMPILE category as well as to reviving the REI/RPC on a new basis. An SQL database persistence back-end would become possible when the Document Generation and Recognition System gets ported from DotNET to ANNA K5.

}

PROBLEMS {

1. There's a corruption of the protected object kind with the cable (ANNA K5, KERNEL) and with it's (KERNEL-ANNA K5) conterpart. The problem occurs after loading the symbolics and cycling once through the persistence file.

2. Extreme performance penalty in map calculations.

3. MaxSize of types are not updated during Grow morphism operations.

4. USES relations must be nameless and their GetName should build a relative path text for display.

}


271 Friday, 24 April 2009

CORE {

An important design goal has now been reached: Memory Defragmentation and Object Relocation.

Of the problems seen in the previous version:

1. The corruption is not seen any more. Merging the kernel leads to some exceptional transient circumstances which confuse the proxies of the tree viewers. Certain cables that the AutoBind creates during the merge could not be deleted. TYPE::SerializeHeap now forcefully clears all proxies before saving and initializes the proxy tree to null when loading. In the layered persistence paradigm, the proxies must have already been saved and destroyed before saving the deeper heap.

2. The performance degradation with map calculations is due to the large reverse cross-reference of scalar types such as Int, Bool, Char, Void. I've added another flScalar flag to TYPE to mark a type as being scalar and should therefore return from GetSize immediately without further deconvolving the type's cross-reference. Scalar are those types that have no relations and their size is determined by the declared MaxSize field. All types start off as scalars and the flag clears once the type gains its first containment relation. The early size calculation algorithm is work in progress so the current remedy to the performance problem is to ease off their XRef by constructing their immediate USES. A performance fix in GetSize with pre-calculating the size of a value containment relation is now using the cached MaxSize value instead of recursing.

3. There were some crashes during the destruction of the system when a scheme of USES relations was in place. USES relations need to cache their size when their target types have been destroyed. The GetSize algorithm promotes the USES types to the rank of fundamental scalar types when their targets are destroyed. This technique perpetuates an undesired dependency on the MaxSize field which could and should be alleviated by generating and demoting clones of the target's Maximum constant. What we are saying here, which is in keeping with the DEMOTION principle of ANNA, is that when a Core is destroyed, the peripheral systems must undertake its responsibilities. Destroying a USES relation differs from destroying a containment relation in that it preserves the size of all dependent types by promoting their containments over to the supplier of the USES. Inserting or deleting USES relations doesn't change the morphology of their dependent types.

4. Initializing a relation has to go through the morphism logic of RELATION::set_Client.

5. Found and fixed a growth problem with USES relations. The growth of a type should only permeate through containment relationships over to the clients of BYVAL relations and not to the types which use those types via USES relations. USES fields inside do not induce size into their container types.

6. Dictionaries differ from object allocators in that their TOKEN is considered to be a dis-owned free-floating used block which re-uses the TypeInfo.Object field to keep information such as InternalFreeSpace. The protection mechanisms of the Morphic Memory Manager now discriminates between dictionaries and object allocators.

7. TODO: MemSize, MemGrow, SelfTestMemory, MemScan and perhaps SerializeHeap are equivalent iterators that differ in their payload only. They need to be reduced to one iterator.

8. Found and removed a senseless self-cable of DISK which hid the inheritance relationship DISK had from its RELATION. Constructing an abstract cell is best done with auto-bind disabled.

The object relocation:

1. Object relocation is provoked by the Morphic Memory Manager when new space is needed and no contiguous block can be found.

2. Object relocation is dealt with in TYPE::MemGrow where used blocks and objects are moved one by-one into their preceding free block neighbour. Our morphic memory manager tends to SEDIMENT used pieces to the bottom of the available space.

3. The algorithm relies on the memory manager's ability to move used blocks while preserving most of their content except TypeInfo.

4. Moving used blocks is a matter which has been resolved by dictionaries for their tokens. An important difference between a TOKEN and an OBJECT is that any pointers buried inside a token is an indirect index through the dictionary's index table which do not require relocation, though the pointer table of a dictionary actually does.

5. A morphic allocator may contain the following:

a. Free fragments of space

b. Used blocks

c. Tokens of a dictionary

d. Object macro-molecules

e. Morphic cells.

f. Tissues

g. Poly-cellular organisms.

6. The relocation algorithm of MemGrow iterates the fragments until it finds the first free block from which to start the defragmentation. It hops over any object or used block in the left-to-right, bottom-up direction of travel from the current TypeInfo->Children field of the allocator.

7. If what follows after a free block is a used block or a token, then a move is done by calling MemRellocate. The move of an used block over a free block has the effect of also moving the free block to follow the used block. The moved free fragment may have been merged with the free fragment that followed the used block before the move, thus yielding a larger free piece.

8. If what follows a free block is an object, then the object has to be moved over the start of the free block. Objects are not linked into the instance list of their allocator but to the instance lists of their type masters. Stepping over an object must therefore be done by calculating the object's size together with the extent of its embedded constant if necessary. On top of an object might lay another object and so forth before another free or used block is again encountered in the allocator. The extent occupied by a train of objects (a tissue) is also known as the difference between two consecutive amorphous objects.

9. Relocating an object temporarily breaks the coalescence of a tissue only to re-form it once the entire compound had been relocated and amorphous matter had again been reached.

10. Before relocating an object note must be taken of its type and physiology class. The relocator temporarily turns the object top container into an amorphous used block inserting it into the heap chains so that MemRelocate may do the amorphic move. The object's typology and physiology is restored once the move is done. STREE::InsertBefore is used to restore an object into its former place among the instances of its type.

11. The object is not operational after an amorphic relocation since all its internal subsystem links retain a content that reflect its former position. Once the move is done, the object needs to be relocated at its new position before the object can be again operational. The relocation of an object's internal state is done in the new OBJECT::Relocated method that every type in the system needs to implement. Another problem is that a certain subset of the object's environment is also invalid until it changes its state to reflect the new position of the relocated object.

12. Every object would also need to implement a method to receive relocation notifications so that it may update its links to a relocated base would need to be implemented, fortunately the core of ANNA uses covalent STREE links that resolve relocating/relocated changes in one single pass over the exact number of objects affected in the environment.

13. Relocation ends with a call to the new OBJECT::Relocated method of the moved object which updates the object's internal pointer map, and also updates the environment reverse cross-reference. Due to some extreme convolution arising in counting and ordering the internal pointer maps of morphic cells such as DISK and KERNEL, a second pass of relocation was required to bind all the STREE parents. A more precise order must be found so that the update can be done in one pass. This relocation rebase algorithm so far avoids using a stack-consuming serializor and rather uses a linearly progressing pointer protection scheme that leaves correct structures at the bottom as it moves up.

14. Object relocation algorithm implements true motion. Moving in space doesn't mean in the real world changing an object's (x,y,z) coordinates as all current programs summarily consider, but is rather a matter of an entity pushing its way through dense matter, avoiding collisions, giving way to others in its place, so as to reach a desired destination.

ANNA's space is viscous (claim)

15. Objects own certain non-object used blocks such as strings for Name, Information, blocks for font data, bendpoints, external constants, etc. Relocating a used block is notified to the container object via the new OBJECT::RelocatedString method vector so that the object can match the block against its pointers and update those affected. Relocating the Name string of a relation, for instance, must be notified to the relation so that it may update both its direct name and that of its connector.

16. The MemGrow routine as well as others are full of pre/post integrity tests and trace messages that greatly helped this development but inflict a performance penalty. All integrity tests should perhaps be wrapped in ASSERT statements so as to be globally disabled. ANNA makes little distinction between a release and a debug version of itself, as it must keep all its symbolics and functionality available regardless. Certain debug logic must be globally disabled/enabled via some flags though.

17. TODO: MemSize, MemGrow, SelfTestMemory, MemScan and perhaps SerializeHeap are equivalent iterators that differ in their payload only. They need to be reduced to one iterator.

18. OBJECT::SetTypeMaster had to change to preserve the object's handle data during a morphism. OBJECT::INIT had to change to clear the handle after SetTypeMaster is called.

Memory defragmentation

1. Defragmentation is now a fairly simple routine behind a new button of the HeapViewer GUI and based on the object relocation. It does the following:

a. It first calculates the theoretical free space in the memory manager

b. Calls the allocator to allocate it in one chunk, thus starving it.

c. Frees the newly acquired block.

A new TYPE::GetSize method iterates the used and free blocks and returns the totals in a point(x,y) structure. When such a call succeeds, the heap is temporarily left starving with no free block. An assumption and integrity test had to be changed in MemRealloc to allow a heap to starve.

2. Defragmentation is carried-out on its own thread while other threads are running. One danger currently lurking in the system is that another thread that runs along with the defragmentor would attempt to allocate memory. Such an attempt would block the contender on the MM spinlock until the defragmentor ends.

3. Defragmentation potentially affects all the objects in the system that lye beyond the first free block. Amongst those are the dynamically allocated THREAD objects that wrap around HOST_OS constructs. ANNA's threads point to the thread handles of the host operating system which, in turn, point back to ANNA's threads via the thread local storage blocks. Moving a THREAD would invalidate the back link from the TLS data. It follows that our THREAD should laso have a pointer to the TLS block so that the THREAD::Relocated routine could update the backlink. The change would add another relation to the Kernel's design and we chose to postpone the move and link the TLS blocks instead. The TLS list can be traversed and the relocated thread can this be found. With this enhancement, ANNA's threads now run while they're being relocated, which is the incipient form of our Metamorphic Physiology. TLS are now STREE constructs that insert one after another in a dis-owned linked list. A protection had to be removed from STREE_BASE::InsertBefore to allow the grouping of orphan children which is potentially dangerous during development as certain tree corruption errors that used to bottle into this test could now go undetected.

4. Defragmentation is an imortant mechanism that we need in order to create compact BSS persistence files. The importance of compacting the BSS is in that it can be re-partitioned for another run of the system. Another benefit is that a compacted BSS represents a coherent design that could be turned into a linkable object in a C++ sense.

}

Schematic Editor {

Changes to support the new object relocation mechanism and protect the schematic editor from the sudden snatch of objects that may be in use. The schematic editor now refers to nodes in the core via proxy objects whose servers may freely be relocated. CurrentNode, CanvasStream and Wiring are now properties whose Get/Set operate on the proxy servers.

}


272 Wednesday, 20 May 2009

CORE {

We are back to a slow but correct version of TYPE::GetSize that doesn't accept trivially the MaxSize field and descends into mapping even for elementary scalar types. The reason here was the dictionary's heap size.

Added a protection to trap decoding errors which yield an out-of-bounds word index. The protection reverts to some saved versions of the word and the encoding stream position so that we may hop around with the debugger and retry the decoding operation.

}


273 Thursday, May 28, 2009 (Milestones)

Several milestones and design goals were almost unexpectedly reached here:

  1. ANNA K5 loads its method bodies from the source code. The core's morphic binary contains most of the source code, minus those model unbound functions. The way is now open to statement-level compilation and to the development of our language and compiler.

  2. The information fields of ANNA's OBJECT can now hold HTML. The result is a vastly improved manner of documenting the system including VML diagrams. Anna's entire concept WEB may now be wrapped around itself into a self-documenting executable. The road is open now for our Tiger2 Morphology Server. The concept must be re-worked with Gaius to obtain XRef equilibrium.

  3. ANNA K5 now loads its MAP file and therefore knows its method entry points. This is an important step towards our RPC.

  4. The new and generalized map-scan based destructor is now in testing.

CORE {

  1. STREAM::GetMapSymbolAddress is a new method that locates a string in a file, moves to the start of the line and reads a pointer value from a hex encoded string it finds there.

  2. A new extension to the file loader now loads a specified Borland MAP file and reads all the method addresses from there. The addresses are then rebased and used by CAT_ABSTRACT::LoadMapFile to build the Global Function Table of the core on Dna[0]. LoadMapFile is a scope recursing algorithm that may be called with a null buffer to calculate the space requirements in advance of calling it to fill the GFT. It may also be called with flag bits 1 and 2 set to list all methods that were bound to code addresses or list those who weren't.

  3. Modified all MODEL.CAT files to emit the ENTER_FRAME(fname) as the EntryCode of all operations. Regenerated the entire source pool from the model. We needed the ENTER_FRAME(fname) macro at the start of each and every method of the core as well as the CLASSNAME definition before the implementation of each class. The macro is needed for a variety of reasons such as building the Virtual Method Description Table. We also want to verify that the virtual method enumerators are well in sync with the method bodies. It might be that two or more such macros should be defined to distinguish between virtual, common and static method implementations. To build the VMT enumerators we must first enhance Gaius to generate them based on an analysis of class inheritance.

  4. We need all manner of RPC to hide behind the ENTER_FRAME macro. It is the place where the call reason should be inspected and an RPC packed must be built based on morphology, then sent over to the endpoint. It may well be that the EXIT_FRAME(fname) will soon be needed as well.

  5. BC502 does not define the __FNAME__ macro and we need one such to trace to the log.

  6. Caught a VMT mapping problem due to OBJECT_BASE::IsEditor not having been listed in the VMT enum.

  7. Added the WCamCore sub-model to the global model of the Morphic Core.

  8. The merging logic was promoted from Gaius to Core. TREE::Merge(src,flags,depth) is a new virtual channel implemented by NODE and LINE to hold the structure merging logic formerly held by Gaius Correlator. The merging algorithm recurses all the way down a scope or line endpoint list with the mgNormal reason to remove each element, then changes to the mgRelocate reason to traverse each element. The EMerge flag is to reuse the same one Merge method and save some design space. The removal before processing is necessary as the merge needs to delete the source as soon as its content was laid over the destination. Without this first step of recursive removal, the deletion of a compound would also destroy pieces that were not processed yet. NODE::SetLineEndpoint and NODE::SetLineEndpoint2 are 2 common methods that together descend, remove and re-point the lines referencing a given node. LINE::Merge does nothing, which means that merging operations have to be started from the point of view of a node.

  9. STREAM::ReadBuf must avoid doing overlapped IO beyond the end-of-file under Windows7, and must test the length of the file.

  10. The Rose loader used to leave garbage in the heap as the objects created did not own their strings at all times. One problem was with SetField whan it was called to replace a string field with the same value. The string got deleted and then recreated in-place. Avoiding. During the load of a model, string fields are owned by the metamorphic RoseObjects which get created to reflect the model. At stage 2, when the rose structure gets parsed to construct our class model, the string fields of those RoseObjects froming the intermediary description structures generally change ownership as they are appropriated by our new classes. The rose intermediary objects must disown them. Added the new XchgInfoString which reads values from the description objects using the serial interface of ReadBuf, and clears the fields to 0 with WriteBuf.

  11. Another important matter is that certain pointer fields of the description objects are other description objects. These must be properly destroyed by recursion. There is now a much better generic destructor in the rose subsystem that aims to become the system's only destructor. The RoseBaseObject destructor takes the instance and launches a map-scan over it testing the fields to determine whether they lye beyond the offset of a base object. Fields that lye beyond are run-time additions to a core base object and cannot therefore be destroyed with the built-in destructors. For each such add-on pointer field a test is done to see if the block is morphic or amorphous, and consequently either the object->Free(*field) gets called for objects or MemFree(*field) for strings and such. Once an instance is cleared of its loosely coupled dependents in its foreign fields, the normal in-built destruction can follow. The mechanism must be proven and then promoted down to the abstract Core.

  12. The Rose model loader was enhanced now with logic to load the UML subsystems. The new CreateModule method function creates constants with the bitmap shape, sets their name to that of the path calculated from the bits and pieces of the CodeFile attribute and the virtual path map macros. The name setting operation results in effectively loading the file's content into the body of the constant node. The shape of the node is then changed to normal shSquare so the constant doesn't flush its content when it gets renamed.

  13. The Rose model loader was also enhanced to relate classes and class categories to their corresponding modules using connectors. The method loader was enhanced to construct a constant with the text of the method as it is now being retrieved from between the begin/end annotations that preserve the method's algorithm. The module is parsed for annotations based on the classname::functionname template.

  14. The end-result of a UML model loading process is that the compiled morphology is complete with function code. This is a long standing mile-stone. The road is clear now for statement-level compilation and to a fully aware ANNA K5 core.

GUI{
  1. InfoView is now a dual tab control which can show an object's Information field in text or HTML. It implements a way of editing html in the old text box and then previewing it in the new WebBrowser control. The viewer is quite powerful now as it allows the editing of node documentation in all TEXT, RTF, and HTML formats. The HTML viewer works as follows: first, any thread that wants to use it must do OleInitialize; when the containing form is created, a navigate command must be issued to a page that sets the styles; When an object is loaded into a viewer, the HTML document is manipulated through the OLE/ActiveX interfaces of the web browser control to grab a handle to the body tag. On that tag, the inner HTML is erased with our ClearBody method and new content is injected for parsing with our AddLine method.

  2. The logic of HTML editing/viewing is also cloned in Gaius Correlator. A new type of a component would be needed here to unify the editing of text, RTF and HTML in a single control. Gaius needs to render large amounts of output and the streaming logic must be promoted to a core stream in tandem with text generation extensions to the existing Serialize methods. The (very old) code path for text rendering is already in the core, and it is working on EditStreams. It also needs to be re-vamped to compile objects off the input stream for symmetry.


274 Monday, June 01, 2009

Renamed TYPE from THierarchy and RELATION from TRelation, as we've been wanting to do for years now.
Encapsulated all the logic that imports a morphology from a UML backbone. The ROSE category is now cleaner, less statics, and the code of the importer is now mapped by ANNA's core. A cleaner model with less duplicated method names. SMapFormatCB is no longer a forward when loading the model. Renamed all the TMatrix::map methods.
strnstr and strnstri have moved from HTML and SCHOOL to HOST where we've got all our extensions to ANSI.
AMORPHOUS has some basic stream capabilities now. It defines the Seek, GetPos, GetSize, IsEnd, Truncate, and Access virtual methods, which may be just enough for binary serialization.

275 Thursday, June 04, 2009 (Milestones)

Another series of painful changes to the core. The most significant development is the growth of constants as their type morphs. There are new extensions to the abstract core to support data inflation and deflation, grow and shrink. Instance morphisms as well as modeling methods as relations of the CODESEG type are both milestones.
  1. TYPE::Grow now carries through the system the data pointer argument. A metamorphosis process would either inject or extract an amount of data equal to the size difference in the morphology multiplied by the overall number of occurrences of the mutation across the topology.
  2. Relations now have a dual purpose: on one hand they (may) describe a rule of injecting substance into their client as described by their supplier; on the other hand the relation field (the dialer) used in the inductor scope for selection purposes is also a type that may associate with other types. In other words, relations are prototypes for other relations. The MS .NET paradigm, like SUN's Java, has a feature that allows attributes to be set on relations. A user/programmer may somewhat extend the language by defining his own attributes. We criticize those attributes as they only extend the system at runtime and they may not be referenced in the design. In our paradigm, the extensions to a relation may be referenced as normal structure member without any lookup. Furthermore, relation prototypes are inheritable. The inductor of a relation may have selection access to the relation's supplier and also gains the substance described by the relation's own fields.
  3. Methods were the first uses of traversal relations, and certain design flaws had to be addressed in the core in order to support them. RELATION::Grow did something that was not compatible with a structure growth, as it rather channeled size changes from suppliers down to clients. This behavior is now called Inflation and it is to say that a relation "inflates" its dependent client type. RELATION now has both Grow and Inflate capabilities to operate on either of the Inductor or the Client directions. The growth of a type is the inflation of its dependent relations. The inflation of a containment relation is the Growth of their client type at the relation's offset, with an amount described by the relation's supplier folded by its cardinality.
  4. Objects relocation was first introduced by the morphic memory defragmentation process where they were shifted to the bottom of the allocation over the adjacent free blocks. Another relocation scenario has arisen now with the new object metamorphosis where an object may need relocation to an arbitrary place in the heap. Object relocation is a temporary death of an object. To be relocated, the object must first be demoted instantly down to an amorphous used block so that it goes back into the heap. Only as either a TOKEN or as an AMORPHOUS used bloc can the TYPE::MemRelocate move the block because morphic objects are not instances of their memory manager. During relocation, objects go back to the heap they came from and they get restored to their proper type right after relocation. TYPE::Reclaim was extended with the keep argument which is now used to do the minimum destruction on the object but not free the remaining used block. The new OBJECT::Grow method invokes these methods to achieve a resizing of an object with possible relocation.
  5. The logic of growing a type by inserting a relation at a place other than the end must be exercised. This is the most general case of structural metamorphosis. The bit shifters of EDICT are ready to do the work in OBJECT::Grow.
  6. Found and fixed an error in TYPE::GetLastRelation where a cable was mistaken for a relation. There was an error in RELATION::GetKind where the scope of the relation was always null and therefore some of the 1-12 containment scenarios described at #Relation typeswhere missed.
  7. RELATION::IsContainment is now used only on traversal in the inflation loop of TYPE::Grow and in the re-linking loop of the destructor for USES. A change to IsContainment so that Auto-definition relations return the supplier.
  8. RELATION::set_Client, RELATION::set_Cardinality and set_Flags are morphic operations that inject data so they should pass through a data handle for uncoiling initialization data. As it is impossible to pre-calculate the space requirements of a morphic operation, the data must be a stream. The impossibility of pre-calculation comes from the irreversible nature of a morphism in the absence of the history band, which is exactly what the data pointer would be. It follows that the data handle must be a STREAM. This is our version of serialization, whereby a system starts-off in its kernel form as instantiated, and grows by a series of morphic serialization injections there-after. Serialization through metamorphosis may lead to the creation of new objects within the substance of a system (cell). All morphic operators need to be enhanced with the stream argument.
  9. If all objects may grow, then so can AMORPHOUS, which should introduce the Grow virtual method prototype. Grow replaces the Truncate of AMORPHOUS.
  10. The new UML model loader reads the method bodies from their source-code files and constructs method relations between the CODESEG type and Char. Methods are modeled as OPERATOR instances so as to get the new spiral bitmap to distinguish them from relations. The creation of methods is a morphic operation that changes the size of the CODESEG type and of its ".code" instance. The progressive growth of .code yields memory fragmentation which eventually triggers a defragmentation. Rose objects, however, are not covalently linked so they are not relocatable. To avoid relocation during the load of a UML model the loader purges its intermediate code as soon as it translated it into the K5 tree. Methods are first wired between CODESEG and Char as empty arrays. The UML loader expands them in a ulterior pass when it reads their text from the module structures. As each method is sized to the cardinality of the text length, the size of the ".code" object grows accordingly. The method bodies are now injected into the growth of the .code constant. The method loader is an excellent exercise of growth metamorphosis of types and their instances.
  11. The UML model loader no longer creates the ENTER, CODE and EXIT constants of each method prototype. The logic is superseded by the metamorphic code segment.
  12. Methods are back to having virtual, static, const attributes, which may finally allow us to build the Virtual Method Tables.
  13. Serializing structures with methods poses now new problems. The serialization of exports is now re-entrant for some reason, and our label protection spinlock was now hit. Had to change the serialization logic so that the spinlock is much tighter around the use of a static buffer, and alloca is used to keep the label within the scope. I am well aware that alloca is going to be a great challenge in the future of morphic stacks.
  14. The UML model loader was taken further. Right after the creation of the loading structure, the Import method now issues a cleanup of the compiler's type loaders and therefore destroys the entire intermediate code. It follows with a memory defragmentation pass. Since our stacks are not yet metamorphic, all variables holding pointers to relocated objects need to be manually reloaded. Reloading may be done by searching the objects by their names to find their new locations and thus reload the pointers. Alternatively, proxy objects may be installed before relocations so as to receive the new locations of their targets. The latter approach is taken for the "gen" pointer the UML loader uses and returns. ScopeViewer and SchematicEditor use proxies to keep track of their current node selections as the heap relocates the objects underneath. A new PROXY_POINTER_DECLARE macro is used in the GUI to add a proxy field and wrap it with get/set properties.
  15. Some experimental logic in TRoseBaseObject::Relocated to map-scan for pointers and attempt to sync them. Extended the RELOCATION_STRING_PTR macro with an explicit object pointer instead of "this" so that traversals may also call it.
  16. Added a direction flag to the LINE::Find(a,b) method so that it may be called from first to last in the method listing logic.
  17. Merging ABSTRACT::STREE with BASE::STREE creates a Promise inheritance relation which is recursive and subject to the RecursionDepth value (of 0). The RELATION::Inflate routine had to adapt to ignore the size impact of deleting such null promises just as it did with the null fractals.
Metamorphosis is confined to objects of type NODE and, of those, only to those whose name starts with '.' until we figure-out a way to protect a type's lower constants from being resized. The int type, for instance, has a full constant MaxInt that determines the type's size, but also has lower constants such as Zero which must not be allowed to grow. The metamorphosis mechanism has many paths and it would take a long time to prove and to mature. Another problem is that the register allocation of the C++ compiler works against us. We would therefore have to step-up the effort of building our compiler with an Intel back-end, or we must try a volatile version of the core.
The system has problems shrinking the .code constant at shutdown.

Other fixes:

  1. Resolved the size calculation for SCHOOL and DICTIONARY. Exceptions for those two types of sub allocators who have extra data as mapped by an 'Extra' relation which needs to be calculated.
  2. Removed the deprecated logic of merging types in Gaius.
  3. Added a flag to Gaius to route the text generation through the Core's serializor. Updated EditStream to send messages to Windows. Added a SendMessage wrapper to BASE.
  4. Found and fixed another scenario where an inheritance relation was skipped in GetLastRelation, because of a cable getting in the way.

Completed another major chunk of work: the Virtual Method Table builder and the EVmt builder. AnalyzeClasses, CountMethods and RCountMethods are the new method analyzer functions that construct the VMT Descriptors, list the methods with parameter lists, and emit a type's specific VMT difference in the form of EVmt enums (milestone). We're a step closer to RPC. TYPE is beginning to use the Unused pointer to int. It holds here the Virtual Method Description Table as composed by our new methods analyzer.


276 Wednesday, July 01, 2009

The following objectives are being pursued:

  1. Tight and comprehensive integrity testing.
  2. RPC
  3. Metamorphic threads
  4. Physiology awareness to be demonstrated by the auto-logger.
  5. Generic destructor
  6. Generic relocation
  7. Fully owned objects, passing the integrity tests at any given time.
  8. Constructions of enum VMTs for all the types in the core.
  9. Rigorous mapping of the class layout and class methods loaded from UML model files over the core's virtual method tables.
  10. Structural metamorphosis involving complex class clusters and instances.
  11. Property access methods as part of the system's DNA.
  12. Better defragmentation.
  13. Model load automation
  14. De-obfuscation of the virtual methods.
  15. Stronger size calculation for relations as they may influence the size of their client, their supplier, their scope or their own. Better traversal algorithms are needed.
  16. Structural metamorphosis must be induced by line insertion/removal.

There were so many changes that I'm having to re-apply one step at the time the differences in order to progressively apply them. The new core breaks the V12 file system compatibility by adding the null constant to the Void type and by correcting a few mapping problems in MORPH and HTML.
Need to set some simpler objectives first and first archive one last V12 file system compatible core, but with as much new functionality as possible.

  1. Want the latest UML loader with automations. One great feature of this automation is that a set of load instructions may be coded as descriptions of the top node of the structure that's being mounted.
  2. Need the latest integrity tester which is much more subtle and caught a swag of problems that have previously gone undetected. One such problem was the insertion of wires into the connectors of relations.
  3. Would like to integrate into the Navigator the new TYPE viewer that would eventually list the DNA of the type.
  4. Integrate as much as possible of the huge number of changes to the model. We should start with the de-obfuscation changes. C++ prototype differentiation by function arguments is not quite supported, so we need different names for methods. All virtual methods must be controlled in the model so that they may be loaded.

So:

  1. De-obfuscating: the relation field value retrieving OBJECT::ReadBuf to ReadBufRel; TREE::FindCreate.
  2. Changed some methods which formerly returned complex structures by value to pass them in and out by reference. This is due to a code generation problem with C++ that passes them as a reference before the "this" pointer instead of after it, so there's no way to reliably get to the "this" pointer when interception is enabled. TYPE::MemSize returning point was an example.
  3. Analyze and CountMethods have moved from Rose.CPP as member of TYPE. Analyze counts the methods in a tree of classes, builds the method lists and binds them to the TypeInfo.Vector of each class. It can also replace the VMT entries of each class VMT with that of proxies that trap into the GenericVirtualFunction. The Analyze method also lists to the logger the differences in the increments in a type's virtual method table as compared to its ancestor. It does so in the enumeration format required by the Core, so that the enums may be copied into the uuids.h file. The core may now use symbolic information describing the layout of its VMTs. The analyzer also warns about the setter/getter methods of the UML model that should be turned into new virtual properties with unique property_Name access methods. It also warns against methods returning non-fundamental types that would have the stack layout problem of pushing the reference after pushing "this".
  4. The UML loader triggers morphisms of types when constructing new numeric constants of types. Enhanced the CAT_ROSE::ReadFieldAttributes to read the options to generate set/get access methods, and their type of storage. Corrected ReadMethodAttribute to flag methods as virtual and static comprehensively. The presence of EntryCode in the model sets the new RELATION::flDebug bit used in auto-logging.
  5. Changed the looks of the debug output. Error message is in red, followed by context data in black. Changed the DBG_ERROR enum.
  6. STREE_BASE needed a destructor. We are now using OBJECT_BASE proxy objects in the core to keep track of pointers whose targets are being relocated during defragmentation. Such proxy objects are allocated onto the stack and they must be properly destroyed when exiting the scope.
  7. TYPE::SelfTestMemory was enhanced to collect the errors encountered while scanning the heap. SELF_TEST_ERROR increments a total_errors counter that may be used to trigger other error reports. All SelfTest methods must now define this variable.
  8. The CAT_ABSTRACT_Type(pivot), and all the macros which build on it, are offsets into the kernel to get from one place to another. The base macro uses GetNearestType to locate the closest result to the caller. GetNearestType must be virtual so that the GUI may also use those macros without linking to the morphic core. During VMT interception, however, this method would also be needed during method prototype de-convolution, and therefore must not be virtual. The solution to this deadlock was to add another FNearestType delegate to the base micro-kernel, wrapped in DEBUG_BASE class, and returning null if the morphic core isn't installed. The core sets its static NearestType function to this delegate.
  9. Needed offset builder macros for STREE, AMORPHOUS, and OBJECT_BASE too.
  10. All VMTs now start with the count entry which the core must correctly set to a vmtMax value for each type. The core will define all EVmt enums as constructed by the DNA Analyze method of TYPE. The EVmt enumerators for the current core have been extracted into UUIDS.H.
  11. STREE_BASE releases the Vector upon destruction. This should resolve the release of all buffers such as TypeInfo, scope, bendpoints, etc.
  12. Enhanced the OBJECT::Grow data injection routine to allow insertion and extraction at offsets other than the end.
  13. The CAT_ROSE::Import method is now more complex and allows a series of post-process operations, depending on the information field of the top node, such as: Kernel Morphology - enable post-processing; AutoDefragment invokes a defragmentation pass; AutoPurge erases the intermediary code of the UML loader; AutoMethods reads the method bodies from their source code; AutoReduceBASE manipulates the BASE category; AutoReduceMORPH erases all constructs that the Core doesn't need; AutoTIV calls a DNA analysis of the loading class cluster and builds the DNA TypeInfoVectors alos generating the EVmt enumerators; AutoMergeBASE loads the BASE category from the model and onto the Core's ABSTRACT branch; AutoMergeABSTRACT loads the ABSTRACT category onto the core's ABSTRACT branch; AutoIntercept switches all VMT entries of the loading classes to the generic interceptor and enters auto-logging mode.
  14. Found a source of errors due to STREE_BASE not initializing the Children and FlatCount fields. We now initialize these fields for the Interlock and Proxies structures.
  15. An effort to correct the memory mapping of the ABSTRACT category which had missing size and black holes. Found and fixed problems in TYPE::MemScan method iterating incompletely. Also found an iteration problem in TYPE::SelfTestMemory when it got to the DISK nucleus which is another constant of the allocator.
  16. New integrity tests: TYPE::SelfTest always calls SelfTestMemory to parse the heap and count the errors it encounters at the block header level. It so can trace disowned used blocks and degenerated (0-sized) blocks.
  17. Added TYPE::SelfTestMorphology which is called on every type master to verify that its format doesn't have overlaps or gaps, and that the size corresponds to the maximum offset. It shows that V12 Core has a layout problem with the HTML::Tags relation, which is why we need to go to V13.
  18. Added the LINE::TestCabling that identifies a series of inconsistencies with lines and cables.
  19. Fixes to LINE::AutoBind and LINE::FindCreate which sometimes inserted wires into relation connectors.
  20. Loading the UML must not create orphan wires. We therefore give it now access to the insertion node. All classes are created and wired under this umbrella. It's an older idea.
  21. Added a grid to the GUI editor of TYPE which lists the virtual methods found in the TypeInfoVector. The viewer lists ordinal, address, type, function name, prototype, and code as retained from loading both model and source code.
  22. Changed the looks of the RELATION bitmap to an arrow with a notch.
  23. Added the flDebug to the flags editor of relations in Schematic Editor.
  24. Created a small BASE.MDL model to test the features of the loader. It maps over the core's first VMTs.
  25. INI_TREE and INI_FILE also have EVmt delta definitions.

Summary of this version:

  1. Has a much stronger integrity tester that highlights a series of problems in the core and compiler.
  2. Builds the method prototypes and correctly loads the code constant using instance metamorphosis.
  3. Builds the EVmts as feedback to the build of the core and sets the lengths of the VMTs. It lists the DNA in the TYPE editor with prototypes and code. (milestone)
  4. it does not relocate the DNA TypeInfo.Vector and therefore incorrectly relocates structures with TIVs loaded.

277 Wednesday, July 01, 2009

Another V12 core was built in the step above. It includes most of the features of this final version.
The essential changes in this version are to do with the layout of the core and with its model mapping.

  1. All property access methods set/get were tunneled through one virtual method called property_FiledName which unifies set/get into one. These are virtual properties that now occupy one place in the DNA vector instead of two. A thorough rethink of all such properties has taken place. OBJECT_BASE::Name is a virtual property with its virtual proprerty_Name(direction,void*data) access method.
  2. All virtual methods hidden throughout the code had to be controlled in the UML modeled so that the VMT's may be mapped correctly.
  3. De-obfuscating identically named methods.
  4. Several point returning methods had to be converted to take the return value by reference. This is to avoid a code generation problem that obscures the "this" pointer when we reach the GenericVirtualFunction.
  5. All Relocated and RelocatedString methods are now controlled into the model, like all virtual methods of the core types. The Relocated methods have been enhanced to cover more aspects of their types.
  6. Added the new Void::null constant to the abstract core, as a place where all trees removed from elsewhere end-up, and also the endpoint of any wire hat gets detached. This is to ensure the highest closure and integrity of our cross-reference.
  7. NODE::Remove inserts the former child into the system's null constant.
  8. A node without a parent is now an error.
  9. All nodes have membrane affinity.
  10. Enhanced NODE::Merge to concatenate the information fields and move the scope and the TypeInfo vector of the source over to the destination.
  11. Destroying a line starts with binding its endpoints to the system's null node.
  12. Relations do not have set_From/set_To accessors, use set_Client/set_Supplier instead.
  13. LINE::set_From and LINE::set_To are the system's principal cause of metamorphosis. It is here that the impact on the size of all parties involved is considered, and inflation operations triggered accordingly. This is an extremely difficult algorithm.
  14. RELATION::GetSizeInfluence is new and calculates the relation's impact onto one of its: client, supplier, field, or scope aspects.
  15. RELATION::Inflate was extremely difficult to get working in this version with all the new scenarios that arouse. Enhanced its prototype with the cardinality parameter. This method hides most details from the user.
  16. Much better logic to trace morphism operations and log those that fail certain integrity conditions.
  17. TYPE::DoRelocation is the beginning of a new relocation algorithm that should work on any structure no matter it does not descend from OBJECT_BASE. The algorithm uses echo-location to inform its neighbors of its new position. The algorithm doesn't work yet and is disabled in the TYPE::MemGrow method.
  18. TYPE::MemGrow must now take special care to update the TIV-DNA vectors of relocating objects, and, conversely, inform the owner of a relocating DNA block. This algorithm too was painful, it is still full of debug code to check the integrity of the environment before and after relocation.
  19. Needed a protection in the relation iterator of TYPE::GetSize against null suppliers.
  20. The callbacks of TYPE::GetSize(mf) return true if they want to instruct the map format iterator to descend the substructure.
  21. HTML::Tags had to be mapped by a relation to avoid a hole.
  22. There were problems with the DICTIONARY::ExtraData and SCHOOL::ExtraData relations. We turned them to statics for now. Dictionaries may not work in this version.
  23. The TYPE::SuperHandle is obsolete, and free to be reused.
  24. There was a hole of 12 bytes in the memory map of ABSTRACT. This is where the 3 non-object fields of MORPH live. The holes had to be wrapped into an amorphous block so that the relocation routine may iterate them. Adding the Globals amorphous header to CAT_ABSTRACT also means introducing the new ptrGlobals relation to map it. This used block has to be initialized before the free heap is and before the entire core, at the top of CAT_MORPH::INIT.
  25. During core construction, there were a few set_Information calls done on objects that already had information strings attached. This resulted in re-allocation and in the creation of several small free blocks. We wanted a core that is invariant to defragmentation as it starts-up.
  26. Included the GuiPeekDispatch back into the OBJECT_BASE::Fork at the place of spin-locking for the child.
  27. OneClearReturnPoint has changed from 12 to 8 bytes, don't understand why.
  28. Using a VVMT.ASM stub module that defines the _ProxyVMTStart label. This module constructs the template VMT interception table. It's a derivative of INTRFACE. The difference is that the stubs call static TYPE::GenericVirtualFunction only to get the method address in EAX, which is going to be jumped upon in a distinct step.

This build has the following major problems:

  1. Merging UML over the ABSTRACT branch leaves garbage cables. The loading relations may be wrong.
  2. RELATION::Inflate reports the shrinking of certain offsets below 0.
  3. Merging the model over the ABSTRACT branch results in corruptions of the type maps.
  4. The new defragmentor doesn't work at all.
  5. The old defragmentor crashes as soon as the structure has TIVs.
  6. Persisting with full symbolic information hangs.


278 Thursday, July 09, 2009

Enhanced the auto-trace feature with function exit interception and with creation/destruction of call frame nodes instances of their methods. The new awareness algorithm works as follows:

  1. GenericEntry gate of the ProxyVMTStart table sets EAX to be the method vector index, then jumps to GenericTail where the proxy VMT stub does most of its work.
  2. Processing of a virtual method call starts with calling TYPE::GenericVirtualFunctionEntry.
  3. The generic function locates the arguments on the stack, gets the "this" pointer, the type of "this" which is the method being intercepted, gets the code address of the method.
  4. If the conditions are right, the generic entry logs the method to the logger, and also creates a node instance of the method which it then returns to the ProxyVMT stub in register ECX along with the target method address in EAX.
  5. Logging is disabled if the method is not flagged for tracing, or if the current thread has any of the re-entry prevention flags set.
  6. If logging is permitted, generic builds a node with the type of the method invoked. The node is grafted at the finger-tip of the thread's scope tree, with the name of the method. The trace message is built with leading blanks to match the depth of the call frame node in the scope of the running thread. Re-entry through the call frame construction logic is disabled until generic returns.
  7. Generic entry carefully returns to the assembler proxy stub with the method's address in EAX and with the call frame pointer in ECX.
  8. If no call frame was returned from GenericVirtual, then the target method can be jumped to with no expectation of any post-processing upon its return.
  9. If a method instance node was constructed, then it must eventually be destroyed, so another call stack frame must be built in order to regain control upon the method's return. The entire last stack frame is copied down, the return address is changed to point to the proxy stub routine.
  10. Execution of the stub resumes after the target method returns. A return value might be expected from the target method in EAX, so this register is preserved.
  11. The stack level is restored.
  12. TYPE::GenericVirtualFunctionExit is now called with register ECX containing the call instance that needs discarding. Generic exit disposes the call node from the thread's tree.
  13. The proxy stub returns to the original caller with EAX preserved as returned from the target method.

The auto-trace is a delicate and powerful mechanism and the basis of our future REI, OTP, FORK and POST. Threads allocate call frame method instances. The current scheme whereby the auto-trace constructs call frames onto the main memory manager is both limiting and dangerous as it interferes with the other threads and with any system objects living there. A thread must therefore either be an allocator in its own right, or be part of an allocator such as a process.

The relocation algorithm. A switch is now programmable to select either the old relocation logic going through the Relocated family of methods, or the new logic going through TYPE::DoRelocation which does it all. The old relocator was corrected in several respects:

  1. A problem in the old relocation logic manifested itself after the TIVs were constructed. OBJECT_BASE::Relocated was attempting to update the TIV entry indexted by the relocating object. It cannot be done that simply, though, and the TIV coherence should be left to TYPE::MemGrow where proper decisions may be taken depending on whether the relocating object is a method or something else.
  2. Relocating a method must update not just one entry of the TIV of the class in which the method is defined, but rather must update the index in the entire DNA of the system core.
  3. Found a really childish mistake I made in NODE::SelfTest where I wrote an assignment instead of a test. I am getting so tired here. The last month was unspeakably hard. The bug corrupted the ClassID right after the integrity tests performed in the relocator.
  4. Line bend points are to be owned by the Connector aspect of a relation and not by the relation's top.

The new relocator was seen working adequately, but not perfectly. Enhancements/corrections:

  1. A protection was needed in TYPE::GetPrev relation iterator to ignore a constant of a type whose bonds have not yet been fully fixed-up .
  2. Map scanning the relocating object to restore its bonds. The callback traps any containments of the STREE aspect and instructs the iterator to not recurse.
  3. Map-scanning the relocating object to update the Handle of its AMORPHOUS aspects.
  4. Map scanning for void pointers, such as node fonts, to also re-claim ownership of such amorphous blocks.
  5. If the relocating object is a thread, then it updates the back links of the thread local storage.
  6. If the relocating object is a node, then the constant handle is fixed-up.
  7. The new relocation method takes an argument for enabling the trace.

The new relocator is slow. It cannot really aim at taking over the previous version and it must have a different purpose which is to generically the user-defined extensions of a core type. The problem with this mixed paradigm of user types with system headers is that both regions must be relocated. The generic relocator, just as the generic destructor, must be called for type maps beyond that of the inbuilt types. This topic must be discussed at the concept level.

Concurrent memory manager.

  1. TYPE::mmm_lock/mmm_unlock are new. They replace the TSpinlock construct with proper threading and interlocking. We must aim to achieve a real-time safe system, most especially the memory management. Only one thread at a time goes through the regions of memory management bracketed by mmm_lock and mmm_unlock. The protected regions were reconsidered. The lock methods use the memory manager as a semaphore which the current thread claims as a whole. There might be a problem here, if the ABSTRACT category also needs to be claimed for any other reason.
  2. Two flags of the THREAD category are now used to prevent re-entry through GenericVirtualFunction. Re-entry needs to be avoided when the memory manager is locked, so bit 1 of the thread flags is used to indicate that the thread owns the memory manager for a protected allocation operation and further allocations should be avoided until the memory manager is unlocked. The former TSpinlock logic was also limited in that it would only work on one allocator, which happened to be the ABSTRACT nucleus. The new logic generalizes to any allocators.
  3. Felt like we should protect the memory manager against re-entry in certain areas of OBJECT::SetTypeMaster, where there's manipulation of the TypeInfo and the VMT of an object so as it may not be parsed in the memory manager by any other thread.
  4. Wrote a no-compromise assembler LockSpinLock which is based on a locked CMPXCHG instruction. Using it in a series of real-time memory management tests with the latest K5 core.

Corrections to the mapping of UML onto the ABSTRACT nucleus.

  1. Chased the metamorphosis problems when merging the Core with its UML. Corrections to RELATION::Inflate to do with setting the cardinality and calculating the shifting offset for those relations that follow the inflating one. Fixed GetSizeInfluence to ignore degenerated relations such as those with null endpoints.
  2. One stubborn morphism error was in TranslatedMoveBy wrongly mapping its matrix argument. Reordered the arguments to get by.
  3. Fixes to the cabling auto-bind mechanism so that there are no cable leftovers after a merge.
  4. Changes to NODE::Merge so that it promotes the wire contents of all cables before resetting their endpoints. During merge, lines are removed with AutoRemove so that the cables get deleted.
  5. A special case of merge for HAS relations had to be taken care of.
  6. Line destructor needs not call AutoRemove, as the operation is done in set_From(null).
  7. Changes to the UML Import routine to merge the ABSTRACT category same way as I do by hand. The block of controlled merging in a specific order for the BASE and ABSTRACT types lead to cycles in the merged result, compounding towards a structure with a corrupt type map. The importer now simply drops the ABSTRACT model over the ABSTRACT node of the core. The result was significantly cleaner.

Auto mounting threads in the GUI tree viewers. All threads are now logged into the Scope Viewer. THREAD::Attach now creates GUI nodes in tree viewer. TK5TreeView::Root is now used to hold all proxy nodes. Root is constructed before the actual windows nodes, so it cannot have a tree node attached. The Handle of the Root proxy node is set to be the TreeView control itself. The Root is therefore invisible, but responds to proxy insertion and removal. Root has a full insertion mask excepting proxy bonds.

THREAD::DONE detaches the thread from the interlock, thus deleting any proxies. The destructor clears the current thread static thread member so that clients no longer rely upon the deleted thread even though it continues to run (down) without our wrapper.

New code in the Gaius GUI frame to format the methods to the RTF output. Generating source code for each method as extracted from the ".code" constant. Method text in blue.


279 Friday, July 10, 2009

Fixes to the relocation logic.

  1. The current thread may be relocated during a defragmentation operation. Locking the memory manager onto a relocating thread needs to be avoided. MemGrow is always called from MemRealloc. MemGrow recursively calls MemRealloc to relocate the current thread. There's no need for the memory manager to re-acquire the lock onto the current thread. MemRealloc has recursion sensing logic and only locks when the flInMemRealloc flag isn't set. This is mandatory in relocating the relocator.
  2. Found and fixed a problem in MemGrow where the TIV wasn't updated if the TIV owner type is not part of the system's DNA. That's the case with user types.
  3. Had to wire another protection in TYPE::GetPrev to ignore relations that have no type. This is the case during relocations when the relation is part of the aggregate being relocated, and its bonds haven't been updated yet.
  4. The new relocator had to be corrected to relocate the handles of any number of constants that may be embedded in an aggregate. Found and fixed a limits checking problem in the logic that fixes-up the bonds of a relocating compound. Found and fixed problems with the ownership of name and font amorphous blocks along containment branches other than the direct ancestry. Logic to prevent snatching string ownership by subsequent referees such as the Connector of a relation might want to do with a shared Name field.
  5. Finally: the Core has certain detached constants for Int, Bool and Char which are amorphous blocks. These cannot be relocated with the generic relocation logic because these types are fundamental, they do not have relations or a format map. The container of such blocks, which is the constant node itself, must get its constant handle updated using the old-style RelocatedString logic.

The new echo-relocation logic is now okay, and has passed the tests, but is unacceptably slow due to the tremendous amount of map scanning it requires, despite the smart case analysis done in CbMapFormatRelocated to relieve the scanner from descending to all leaves of a structure. In contrast, the specific relocation methods have inbuilt knowledge of their relocation offsets upon which they operate the fix-ups straight-on. This is the very trade-off between generality and specialty in terms of handling effort. It feels like a corolary that smarter learning and evolving systems must run slower than their static counterparts.


280 Friday, July 24, 2009 (interlocking)

Re-worked the real-time interlocking system. Let's remember why we got here: once the auto-trace activated, the system starts furiously allocating memory for the call frames. Stack is a push/pop device traditionally much simpler than the heap. Memory management became concurrent as the threads of ANNA were all auto-tracing. Memory corruptions and system crashes were immediate. At first the mmm_lock/mmm_unlock seemed to be the problem in not being placed to protect all critical regions of memory management, then a deeper problem with the interlocking system itself was found and had to be addressed.

  1. Developed and then progressively refined a console test for real-time interlocking. The test now has 7 branches as declared by the ABSTRACT::ETestSuite enumerator and passed through ABSTRACT::TestSuite to THREAD::TestThreads, all meant to exercise the THREAD-OBJECT interlocking mechanisms. The more complex test does the following:
    1. Forks a test process with a series of arguments: number of child threads, test duration, internal pause after some major test steps taken by the child tester.
    2. The test process constructs a semaphore and claims it, forks the specified number of child threads which will fight for resources, waits for the threads to start running, releases the semaphore, waits for all threads to exit, checks some over-all integrity after those threads have run, destroys the semaphore and terminates.
    3. The child threads do different things depending on the test number invoked for the specified amount of time, then exit.
    4. The simplest test thread attempts to claim the semaphore and normally blocks as the semaphore is owned by the master test process. When the master unlocks the semaphore, the core's interlocking system picks-up the first interlocked child thread that is blocked onto the semaphore and resumes it as the new owner of the semaphore. The other threads remain blocked. The simple test thread unlocks the semaphore and exits, thus allowing the other blocked threads to run and do the same.
    5. A step up from this test is one where the child test threads lock, sleep, and unlock the semaphore for the specified test duration.
    6. The main objective of interlocking is to protect memory allocation and tree insertion. The following tests would stress those features.
    7. A more complex test creates a maximum number of node constants of either constant or varying size and inserts them into the thread node where it later finds them to decimate them. The thread has programmable pauses after creation and destruction of nodes.
    8. The final test is aimed at stressing the memory manager as the semaphore so the test threads run with no specific synchronization to create and destroy node constants to fill and empty their buckets.
    9. One more test runs all of the others in a loop.
  2. As the interlocking system was totally broken, a swag of new real-time monitoring features had to be devised to catch the problems.
    1. The following strategic counters were added to the design to measure the real-time integrity:
      OBJECT::locks, OBJECT::unlocks, TYPE::locks, TYPE::unlocks, OBJECT::OverlockCount, OBJECT::interlocks, STREAM::opened, STREAM::closed, THREAD::suspends, THREAD::resumes, THREAD::interlocks, THREAD::RunningThreads, STREE::lock_count.
    2. The new OBJECT::CheckClosures method checks the equilibrium of those counters and reports errors if something is not in balance. The object locks-unlocks cannot be negative, for instance, as it would happen if the user unlocked resources more often than it locked them. Another error, which is more complex to detect, is to have the unlock expectation exceed the number of threads running in the system.
    3. The closure monitor is placed strategically at count modification points and at interlock insertion points. Calling this method out-of context would yield transient errors. The main test process has been calling it while waiting for its children, but not anymore. To avoid fluttering errors, the closure monitor checks each condition twice before retaining the error.
    4. Core is constructed on the init thread inherited from the operating system which launched the ANNA K5 process. We do not wrap this thread, and the interlocking closure during construction/destruction is out of balance as a result. We fake a running thread count during INIT and DONE of the core to keep the balance.
    5. The closure analyzer was falling out of balance due to the core's Autoload stream which stayed opened until the destruction of the core, but without interlocking with the init thread, since the init is not controlled by ANNA. We're now closing the stream in INIT, and re-opening it in DONE to preserve the interlocking balance.
    6. The main test process displays the interlocking structure once every second while waiting for its children to return. This feature catches situations such as threads locking themselves through recursion, or they vanish from the micro-kernel's interlocking structure with no chance to return.
    7. There's a GUI version of the closure monitor which runs on a 100ms timer of the editor for type ABSTRACT. The GUI displays the counters in silver when they're in balance, or in red otherwise. The GUI presents the controls to select and configure a test in terms of test name, test duration, inner pause, and number of threads to spawn.
    8. One major difference between the console test and the GUI version, is the influence of GUI proxies over the test. Each update of the interlock changes the tree view, adding another layer of complexity over the matter. While interlocking, the core is allowed to be re-entered with traversals of any tree aspects as may be required by GUI navigation.
  3. Tremendous effort went into testing and correcting this version of the interlock. A summary follows:
    1. Any increment and decrement has to be locked to be atomic on my multi-core processor. The LockedIncrement and LockedDecrement functions are there for the job, implementing an inline assembler LOCK INC instruction. No closure monitoring could be taken seriously without those locking functions.
    2. Seriously wrong was our thread handle which was in fact a pseudo-handle. As the Windows API states, the pseudo-handle is only good for work on the current thread but cannot be used to identify other threads in the system. Our THREAD::Suspend always worked since ANNA only ever suspends the current thread as it locks onto an already locked resource. Our THREAD::Resume never worked, though, since an unlocking thread must awaken another, and the handle is wrong (in the -1, -2 range). Added the new void*HOST_OS_RT::GetHandle(src pid, void*handle) method to do process manipulation and return an inter-thread usable handle. GetHandle was implemented in the REI INTERFACE. THREAD::Resume now works.
    3. Another major problem and ongoing source of concern is that Windows does not allow a developer to subclass the dispatcher, which is exactly what we have to do in the core of ANNA. The problem here is that we cannot atomically insert the blocking thread into an interlocking tree and suspend the thread atomically. One cannot really be at the suspend point, but rather on the way down to suspending, and back again transparently when another thread resumes this one. We needed a last-moment indication that the current thread is really suspended, much in the way of a flag set in our thread structure. We would then only pick truly suspended threads from the interlock tree, the others being on their way to suspending. The OBJECT::Lock function cannot be spinlocked against all other threads across a suspend call, naturally. So, what we settled for was to port the flag field all the way down through the layers as if windows SuspendThread took such an argument, and set it as the very last operation before calling the API. I assume that the API would resolve in a limited number of instructions, less than the distance from our OBJECT::Unlock spinlock, past Interlock rearrangements, and down to the Resume call.
    4. The ultimate interlock owner is the uKernel where threads bind when they aren't blocked. The new and corrected containment scheme now promotes the VLock/VUnlock as well as Change and property_Name calls all the way to the uKernel which may be masqueraded by Form proxies which are disowned. Had to add a protection in UPDATE_PROXIES against the disowned proxy. TNotifyRelation of Forms did not override property_Name resulting thus in a core reset of the name.
    5. Threads attach to the micro kernel so they always have an interlocking parenthood, whereas resources do not. OBJECT::Lock checks that the locker thread and the locking object have no common parent for a successful claim, or the uKernel as their common parent when the resource is already locked by another thread. OBJECT::OverlockCount count is incremented when the same thread locks onto a resource twice. GetCommonParent is therefore the workhorse of the interlock and was enhanced with a new parameter to specify the maximum depth of search beyond which the tree should be considered to have formed cycles. STREE::GetCommonParent only allows 9 degrees of separation in a tree, plotting at limit the parenthood chain in the vicinity of a possible cycle. There was a problem with this code as it tried to match the root MicroKernel as an STREE when in fact the uKernel is an object, so the function never found the common root and sometimes returned errant common parents.
    6. OBJECT::Lock and OBJECT::Unlock are the system's dispatcher. A thread gets to Lock when it wants exclusive access to the object. If the thread locks an object recursively the objects OverlockCount is simply incremented only to be decremented by the pairing Unlock of the recursion's backtracking path. The object's Lock routine, however, may have already been entered by another thread which did not yet exit. Two threads running the Lock/Unlock code for the same object is an ambiguity that would lead to the corruption of the interlock tree and to THREAD::Suspend/Resume confusion. Access to OBJECT::Lock/Unlock is therefore protected by spinlocks over two flags of the object, flInLock and flInUnlock, and is granted only when both flags are clear. The spinlocks wait for both flags to clear and then set one of them all in one atomic operation. Once a thread gains access to the object's Lock or Unlock logic, it is guaranteed exclusive access to the object's interlocking structure. One potential problem may be that the interlocking tree may need to be recursively spin-locked upwards towards the MicroKernel's root. In our tests we haven't yet created or encountered this scenario. Such would be the case with a deeper interlocking tree. The current thread locks the object by binding it into its own interlock structure where the thread links all its exclusively used resources. There are normal cases when the resource object has already been claimed by another thread. The current locking thread then needs to graft itself with all its assets into the interlock of the resource object and suspend itself. A future Unlock done onto the object by its owner thread would remove the first suspended thread from the object's interlock and restore it to the MicroKernel's run list, then resume the locking thread which would now find the object's interlock free for it to claim. Unlock only picks those threads that have the flBlocked set and flInLock clear. At the last moment before suspending, the current locking thread must release the object's flInLock spinlock so that the other releasing thread that owns the object may enter the Unlock routine. Lock and Unlock manipulate the OBJECT::interlocks and THREAD::interlocks global counters. The object interlocks counter goes up when an object is successfully grafted to a thread's interlocking tree, and goes down when the thread removes the object. The thread interlocks counter goes up when a thread blocks into the object's interlocking tree, and down when it is resumed by Unlock. A check is done at the end of Lock to verify that the current thread is indeed the new object's owner and trap a system failure otherwise. Lock and Unlock clear the flInLock or flInUnlock from the resource's flags upon return. Lock and Unlock issue VLock and VUnlock proxy notifications as well as sends Change(ecInterlock) messages towards the root. Change notifications, as a reminder, are used in calculating the dirty state through-out the system.
    7. Reviewing this Lock/Unlock code makes me feel (again) that holding per resource InLock and InUnlock spin flags might actually be wrong, and concurrent thread accesses to the interlock in general could lead to corruption of the interlock tree. To test with a global dispatcher flag that serializes access through Lock/Unlock requires a stable system secured into the archives for comparison. Per resource spinning would prove much better with large schemes where fewer threads contend over many different resources. Spinning them all in one point would be wasteful.
    8. Entering critical regions of code is protected by spinlocks tested and set atomically by LockSpinlock. Each spinlock is a char value.
    9. OBJECT has a Flags field now. The new flInLock and flInUnlock flag bits that need setting in thread exclusive mode with the new OBJECT::SpinlockInterlock method. You should see the Lock/Unlock as a massive multi-level spin, on narrowing orbits, funneling down to suspend and resume. On the wide orbit of SpinlockInterlock, the current thread dispatches out if the entry condition is not met. The flDebug flag was generalized to OBJECT.
    10. Blocked threads are identified by their flBlocked flag set.
    11. Two new common methods TLock and TUnlock hide the details of Lock/Unlock for the most common use case.
    12. The amount of spinning is recorded in phase fields that should one day be used to calculate the time a system wastes synchronizing.
  4. THREAD changes
    1. THREAD is now a TYPE descendent, not a STREAM. The STREAM class is now clear of all complicity to threading. The move is in preparation of turning each thread into a memory manager for method call frames, and for merging STREAM into OBJECT or such so that all core objects gain stream capabilities. STREAM::ProgressIndicator was promoted to OBJECT, although it should vanish and replaced by the more serious proxy logic.
    2. THREAD::ID is an internal unique value that K5 assigns to each thread when it's created. The value is ever incrementing. A new thread GetShortName returns a modulo 10 reminder of this ID as a 0 to 9 digit to be used as cryptic fast logging of THREAD::Suspend and Resume.
    3. The new THREAD::INIT_THREAD and THREAD::Free are of extreme difficulty as they need to create/destroy the thread instances in the memory manager, without interlocking with it for memory allocation.
    4. THREAD::get_Current may, at limits, return null. Needed some protections such as in ABSTRACT::DestroyThread.
    5. When created, threads are inserted into the scope of UserGroup.
    6. Gone back to the former TLS scheme where TLS blocks are not linked in a tree. Each THREAD now has a Tls pointer used during relocations. The tree-chaining of TLSs was troublesome and the cause of certain crashes.
    7. THREAD adds several counters for closure analysis. The total suspends and resumes counters which must obey some closure rules: a system cannot have more resumes than suspends; the difference between suspends and resumes which we call expectation cannot exceed the total number of threads in the system. The thread interlocks counter reflects the number of threads that have been blocked onto resources. The running threads counter reflects the number of threads currently running user code. Running threads counter gets incremented in the thread constructor as the thread makes its way from Fork into user space. The issue of constructing threads is complicated, so is the destruction.
    8. OBJECT_BASE::Fork uses a tight LockSpinlock to avoid re-entry. Fork needs serial access to share the spinlock with the child and wait for its launch. This state cannot currently be shared with other forking threads. No sleeping in Fork. The child forked thread releases the parent forker thread as soon as it had copied a stack frame of the parent's context. The copy is taken upon return from the thread constructor. The child must eventually live on the heap. The new threads are memory managers that will eventually hold the entire morphic stack, but until then, they must be able to allocate auto-trace call frames to be used in interlocking. Those memory managers must be resize-able and relocate-able, therefore live onto another metamorphic heap. Thread contention occurs as soon as calls through the core to allocate memory are made and hence interlocking begins. Interlocking is not possible in the absence of our not yet allocated thread instances, which is where a vicious circle closes. To resolve the matter, we first construct the thread onto the system-controlled stack forked to us by our parent within a chunk taken by alloca, of a size calculated from the constructing type, as we may be building a thread descendent. The stack-resident thread is configured with everything it needs to run, including a real host OS handle, and it is inserted into the null interlock tree through a call to Attach. The thread is both in the interlock structures as well as is bound into the TypeInfo registrar of THREAD type. The running threads counter is incremented, the stack limits are set, etc. The stack-resident thread is capable of non auto-traced execution through the core, so the flInTrace flag is set. It can now issue memory allocation calls and blocking calls through core constructors. The stack-resident thread gets a free block of memory from the core allocator, and intends to clone itself there. The used block has the invisible amorphous header which needs special handling, but he entire thread content can be copied over to the raw block. Certain header bits such as the handle and the VMT are copyied from stack over to the used block, eventually getting two almost identical copies of the thread. The current stack-thread is removed from its type-master just as the raw copy gets inserted. Up to this point no other thread would interlock the constructing one, so these manipulations may be quite safe. There's the problem now: the clone is not in place as the environment all points back to the original stack variant. Here's where ANNA's crucial relocation mecanism comes into play to do the update. When relocation ends, the original and the clone have swapped places and the stack-block is obsolete and would vanish out of scope along with the constructor frame. The last thing to do is to set the thread's name, an issue carefully postponed so as to not interlock too soon with the stack-resident thread while it was being built. The inTrace flag can now be cleared, and auto-tracing may commence. The new heap-resident thread would return to Fork where it gets some final touches and would continue to run the forker's code there-after until its end of life. The whole construction process is described in some 40 instructions where order is paramount.
    9. At present ANNA's core has several ways to free objects, yielding some confusion. Heap objects can be released with a call to the DONE destructor, followed by a call to MemFree or MemRealloc. OBJECT::Free does this succession after locating the object's top following the containment indicators. Freing a heap-resident thread poses the same issues as constructing them, and better control of the steps involved is required. THREAD::Free overrides the core's basic destructor/de-allocator. The special destruction logic is only valid for destroying the current thread. Destruction of another thread under the control of the current thread is an unknown, and currently reverts to the base Free routine.
    10. Deleting the current thread. The one last safe memory operation the heap-resident thread can do is to release its name. Any trace thereafter relies solely on ID. This release is also the last auto-traceable function call, after which the InTrace flag needs to be set. We now want to clone the heap-resident thread onto the stack and relocate it there. The somewhat more difficult aspects here as opposed to constructing is that the running thread surely has proxies, interlocks and children in its scope. These structures must be copied and relocated, and not destroyed as a normal destructor would do. In fact we re-construct here the original after the copy was taken. Once the stack-resident copy is activated, the original heap-block is reclaimed. Reclaiming is a blocking operation through the core, and collisions attempts are expected for handling in the interlock and the stack-resident thread is there to support them. Once reclaiming the original returns, there can be no other blocking operations on this thread. We unregister all aspects of the current thread from the system with a call to the DONE destructor, and decrement the running threads counter. The thread is now stripped of our wrapper class and will run (exit) back into the operating system for the host-OS side of the destruction.
    11. Another matter of great difficulty with ANNA's thread is the logic of Suspend and Resume. Dispatching operations currently leave 3 chars trace messages representing the char id of the current thread ('0'+ID%10) the car id of the suspending thread and pipe to indicate blocking. Blocking is currently only ever done by the current thread upon itself. Resume echoes the char id of the current thread, the char id of the awakened thread, followed by underscore to indicate passage, A run sequence would plot something like 11|23_21_33|13_ meaning that thread 1 blocked, thread 2 resumed thread 3 and then 1, thread 3 blocked but was then resumed by thread 1. The thread suspends and resumes counters are updated and checked with the rest of the system for closure. Some end-game errors are also trapped here. There's a difference now between suspend and resume, in that suspend may actually be called with a timeout value, if the OS supports it. Windows doesn't allow resuming a sleeping thread, unfortunately. If suspend is called with timeout, the thread sleeps for that amount, no-matter the resumes. Timeout is required by the Schematic Editor as a way the share a node with another mission critical serializing process. The thread is flagged as Blocked whether sleeping or suspending. When sleeping, the flag bit isn't set, so the unlock routine does not pick the sleeping thread from the resource's interlock. The low-level host OS suspend API may fail, which is catastrophic. Resume, on the other hand, may need to be issued several times for the host to get the command.
  5. Changes to TYPE memory manager:
    1. Added the TYPE::locks and TYPE::unlocks global counters for closure monitoring. Added closure error checking at mmm_lock and mmm_unlock. The memory lock/unlock routes through the new and simpler OBJECT::TLock and TUnlock. The memory manager sets and resets a THREAD::flMemoryLock flag to prevent auto-tracing the auto-tracer.
    2. The global locking of TYPE::MemScan was disabled as it causes more deadlocks than protects.
    3. TYPE::Reclaim was revised to avoid virtual method calls, thus survive the auto-tracer. The method also has an unsafe region where an object's neighbors are found and the object is promoted to a used block all of which had to be locked. Moved the locks around the code. Reviewed the locking in MemRealloc.
    4. THREAD::flInTrace is manipulated in TYPE::GenericVirtualFunctionEntry and Exit who had to take into account their possible recursion. The flag was not restored when auto-trace exited.
    5. The GUI thread now runs with auto-trace disabled so that the main thread doesn't get hogged.
    6. TYPE::CreateInstance could theoretically be ordered so that it doesn't need any memory locking. Without a memory lock, however, the tests crash significantly more often, probably indicating another problem elsewhere. The same is true for OBJECT::Free, where only the DONE destructor should be protected with the memory lock, and lot the entire DONE-Reclaim sequence. We keep the locks.
  6. Changes to OBJECT:
    1. OBJECT::property_Name wrongly issued proxy updates on the set_Name function.
    2. OBJECT::SetTypeMaster had to synchronize with the system's global memory manager and not with the local container of a constructing/destructing object. The memory locks in this code were also repositioned.
  7. STREE operations are mostly protected now with a correct thread id and with air-tight locked spinlocks. Raw, unlocking variants of Insert and remove are provided to be used in code which is already protected and the spin would otherwise lead to deadlocks. STREE::lock_count measures the closure of locking onto the core's low-level tree bonds.
  8. Certain virtual methods need to be called normally to avoid auto-tracing them. Certain methods such as VLock and VUnlock may crash the auto-trace and must never have flDebug set.
  9. AMORPHOUS::IsObject does loops once through its logic to check the stability of its result and thus avoid real-time flutter.
  10. Incredibly, the construction of the core was all wrong as the meaning of the flNotContained flag was in fact the opposite. Logic such as OBJECT::Free did not climb up the containment enough to locate an object's top, and only partly destructed structures, then partly de-allocated them, can't understand how we survived for so long... Correcting the bit revealed a series of other problems we thought had gone away many years ago. Destruction of the core entered endless recursive loops with no much hope of resolving them. Fortunately, all recursions had the intertwined lines as the single cause. Re-worked all category destructors to first destroy registered relations, then unbind any lines from the object being destroyed, then destroy the remaining type-masters of the category. Core destruction now succeeds. We must investigate again the node merging logic that was so troublesome a few revisions back, as the orphaned lines remaining then may actually have been half destroyed relations, as in fact we suspected. OBJECT_BASE::flNotContained renamed to flNotOwned and it now represents the bit position. NODE::Relocated and others now properly iterate the containment to re-base strings such as name and font.
  11. Caught a time calculation problem in HOST_OS_RT::SetTimeReference, where calculations had to be done in the __int64 domain. Large time elapses were probably wrong, with consequences, perhaps on the lifetime of our licenses.
  12. HOST_OS_RT::Fault finally logs the message before breaking into the debugger.
  13. Repaired the VLock and VUnlock of TIsNodeClient to also update the label of the tree node to include the "locked by" annotation. TIsNodeClient::property_Name was wrongly conducing the call through and back to the BASE core for write, resulting in the over-set of the node's name, using a memory-(re)locking strdup. This path was also invalidating a relation's cable name, which would lead to memory corruption when the relation got destroyed. The tree viewers are now quite responsive to all manner of changes to the interlock.
  14. There was another proxy update problem due to the duplication of ofChangeNotifications options flag both at OBJECT_BASE, and wrongly at OBJECT level unexpectedly disabling the tree view updates.
  15. A fix to TREE::FindCreateUML to allow finding a direct child without name separation conventions.
  16. The main thread of each GUI form is flagged as such with THREAD::flGUI. This thread should perhaps not really be blocked... although, right now, all threads process messages.
  17. Added the NODE::ofOrphanRemove flag to allow the interlocking test to avoid the autobind mechanism which tends to corrupt the chains of the null constant during the tests. Autobind should perhaps be invoked only if the node is seen to have an IO interface.
  18. The sanity check made at the end of Lock to see if the lock was acquired failed for selections in the schematic editor if one picked a line in the ABSTRACT category, then bent the line to provoke memory re-allocation. The issue here is with indirect locks. The check was too simple, so the stronger IsLocked is now used.

281 Wednesday, July 29, 2009

THREAD cloned the STREAM archive and split away in preparation for hierarchy changes.
The major design change in this version is that TYPE descends from STREAM. The consequences are that every type in the system may be a memory manager as well as a stream, file or socket, simultaneously.
Moved MaxSize from NODE to AMORPHOUS as we've been threatening we would for some time.
Added the TYPE::SizeCache field. SizeCache is altered by TYPE::Grow metamorphosis operations.
Overlocking doesn't post any change or VLock notifications.
THREAD::Free method was not mapped by the UML model.
Added the THREAD::MemoryManager field which gets initialised to AbstractKernel.
CreationTime and LastModified have been promoted from OtpBaseObject to OBJECT_BASE as the core begins to need date and time stamps.
STREAM has a new flOpened flag which gets set by STREAM::Open with a mode other than -2. The flag gets cleared by STREAM::Close. A stream which doesn't have the flOpened bit set does not attempt to close its handle. Since every TYPE instance in the system is now a STREAM, the constructors must avoid auto-opening the handles by specifying an open mode of -1 or -2. Open is called unavoidably from the stream constructor and Close from the destructor. The two special modes instruct the Open to either return immediately (-2) or to lock and return. One important mechanism for streams is the auto-locking on construction. Using these modes, objects would be free to give heir handle pointer any interpretation they wish. A constant, for instance, would have a non-void handle which is not a file handle.
Added a protection to property_From and property_To against treating endpoints which were set to K5NULL as being types.
OBJECT_BASE has its own Flags which is now separate from the Visibility field.
We seem to have too many Flags fields. Reduced the flag fields of OBJECT with those of OBJECT_BASE.
TREE::EDirection was the source of a major confusion when it clashed with AMORPHOUS::EDirection and anti-aliased methods with such arguments. We obsoleted TREE::EDirection, although we still use its FROM and TO constants, remaining to demonstrate why a line's from and to endpoints are the same as reading and writing.
The relocator had problems now relocating the current thread. A new object flRelocating flag was added to inform the system of which is the currently relocating object. The system would have to treat such an object as invalid, and try to either avoid it or block for it to become valid.
OBJECT serializes the Flags field and Visibility separately now.
Dictionaries introduced a nasty disequilibrium in that their TOKEN structures packed so hard over the amorphous used block, with re-defining the TypeInfo.Object pointer as a union as several other fields. Tokens had to be considered to not be contained. It is why a user of a token could not retrieve the dictionary. All those special fields of TOKEN are now explicit fields of AMORPHOUS. Every block and object in the system now has the internal free space field, a Crc8, a flag field of 8 bits, and a LCountSz which I currently don't know how to generalize beyond dictionaries. TYPE::MemRealloc now preserves the block's container pointer across a relocation call.
The count of entries defined in the vvmt.asm module was to low to map the system's vmts. Increased the length from 256 to 1024 interceptor entries. This was the cause of a failure in auto-trace that was difficult to trace.
We are constantly reviewing our enumerators in order to turn them all to the (1<<flagID) scheme.
TYPE::CreateInstance has to set the handle and MaxSize of the new constant just right after claiming it as the type's instance. Having had used blocks on the heap with handles set to something other than their size was a transient dangerous condition for any other traversal.
Some disambiguation of methods.
Canvases must also be nested into some scope, otherwise the integrity tests fail. Canvases are now born in the HOME folder.

This version has been checked to do the following:

  1. Construct & destruct the core from the command line.
  2. Load the core's model and turn on the auto-interception of method calls.
  3. Log the entire thread of loading Gaius with content.
  4. Log the entire progress of an 4-threaded LockedMemoryAccessVarisize real-time test for at least 2 hours safely.
  5. Defragment in various stages, multiple times over, full core structures with TIVs.
  6. Serialize out and in again the demo model.
  7. Persist with a model loaded.
  8. Runs the ThreadLoopAll for 1/4H before dropping-out with a transient closure error.
  9. Interlocking with the Schematic Editor seems to work.

Problems:

  1. The real-time LockedMemoryAccess threading test hangs when run from the console.
  2. The real-time ThreadsContendingOverMemoryVarisize threading test deadlocks.
  3. We yet cannot serialize or persist structures with TIVs.
  4. Auto-trace hangs upon the destruction of our real-time test threads.
  5. STREE is 48 bytes now and RELATION has bloated to 537 bytes.
  6. Traversing the Interlock while threads are in conflict is dangerous and needs spin-locks.
  7. MapScan is even slower now, more so with full symbolic information loaded.

Perhaps V281 is not too bad...


282 Thursday, July 30, 2009

THREAD initializes its system stack into a memory manager. At the bottom/start it inserts a used block followed by the empty space of the stack. The stack size is a configuration argument passed through Fork. THREAD's memory space spans between the addresses returned by the new GetStartAddress and GetEndAddress which coincide with the stack's TLS values recorded during thread creation by SetStackLimit. The stack's bottom is calculated by presuming the presence of a certain amount of system call frames whose dimension depend on the OS and on the begin_thread runtime library function. Under Windows a 64K area is subtracted from any stack size. A thread forked with a stack of 0 will be set to a default 1MB stack size. The stack limits are now set at the beginning of DoStartForkThread as the very first operation, and before creating the thread wrapper over the system thread handle. Since threads now have a span of free space, their editor is beginning to show the memory map control.
Changed/fixed the MemScan routine to show the block of a heap that doesn't start-off with an object.


283 Tuesday, August 4, 2009 (milestone)

We've reached an important milestone: ANNA K5 is now aware of the morphology of its threads to the level of its intercepted calls. It now constructs and links nodes for every intercept-able call that a thread makes. The auto-tracer of this version no longer clones the last call frame of the intercepted method, thus preserving the identity principle that must govern the system. Instead, the new auto-tracer wraps a node structure around each call-frame that it intercepts. As a call is traced, the thread's environment is parsed to see if the method is already being used in this context and a new USE relation is created if not. Each new call frame is then attributed as instance of such a USE relationship. So, yes, USE relations may have instances. The USE instances are indirect instances of the used method prototypes and therefore have a size and a layout to cover the formal arguments list of the call. Those method instances are also constants, in that they also present an amorphous tail to cover the extent between the latest call frame and the previous, as ordered onto the thread's stack.

  1. THREAD constructor and destructor were re-worked to make a usable heap out of the thread's stack. To that effect, the constructor creates a used block at the bottom of the stack followed by a free block to cover the remaining space to the top. Call frames will evolve from the thread's stack top as instances of the use relationships that a thread makes with its environment as it runs. Those nodes are managed by auto-trace through the assembler shim GenericEntry that calls TYPE::GenericVirtualFunctionEntry and Exit. The bottom of a thread's stack can be used as a normal heap. If instances of the thread's uses relations appear here, they would be interpreted as Inter Thread Procedure Calls, and would get executed in the context of the container thread.
  2. We've reinstated the ENTER_FRAME macro which now allocates a 256 byte buffer at the very top of the call frame, before any explicit local variables, to build here the call frame method instance that maps the stack to Anna's metamorphic core. The compiler used to build ANNA K5 must disable register variables to keep the precise nature of a function's local arguments. Borland compiler is now configured with #pragma option -r-. There's a rather subtle interaction between TYPE::GenericVirtualFunctionEntry and GenericVirtualFunctionExit driven by the GenericEntry assembler gates, one that requires the transmission or the preservation of registers EAX, ECX and EDX. ENTER_FRAME adds reg_EAX, reg_EBX, reg_ECX and reg_EDX to any call frame so that GenericVirtualFunctionEntry may transmit those registers over to GenericVirtualFunctionExit.
  3. The GenericVirtualFunctionEntry logic creates a node for the call frame in its own local space and returns to the GenericEntry without destroying the node. GenericEntry calls the real function, which then finds a pre-constructed call frame as its first local variable. It too must preserve the structure past the end. Normal code must not presume the layout or content of their un-mapped local space. GenericEntry calls GenericVirtualFunctionExit which also finds the call frame as its first local variable to destroy. As the last step of auto-tracing, GenericEntry jumps to the return address preserved in by GenericVirtualFunctionEntry and Exit in reg_EDX.
  4. The map scanner TYPE::GetSize was enhanced to ignore any constants from the size calculations of USE relations.
  5. Added TYPE::auto_trace_IO closure monitor to check the balance of GenericVirtual Entry and Exit.
  6. TYPE::Reclaim needed a fix when deleting an object at the end of the heap to chain it tothe first non-object block found from the beginning. Before the introduction of call frames, we never had an object instance right at the end of the heap.
  7. Objects are now contained by their memory managers and have their TypeInfo.Object field set to the allocator. Free floating objects are not permitted and they provoke errors with the integrity test. The only two disowned objects are the operating system itself and its null constant which is the base micro kernel. The flDisowned flag is set for those portions of an aggregation that may be freed independently from its neighbors. This move is in support of multiple memory managers. TYPE::CreateInstance uses the allocator configured for the currently running thread. Same does OBJECT::SetTypeMaster, mmm_malloc, mmm_free, mmm_realloc, and mmm_memreset. OBJECT::FRee, however, determines the allocator used to create the destroying object from its TypeInfo.Object field.
  8. NODE::flAutoWeight is new and the logic behind the flag is to have a node constantly follow the weight center of all other nodes to which it relates via connecting lines. We plan on allowing THREAD nodes to migrate through the scopes to follow their relatives and thus help determine the limits of a thread's processing cone. Such migration would be followed by a corresponding visual effect of seeing the thread move across the schematic towards, and in the middle of, the method prototypes it so far invoked. NODE::AutoWeight is a new method that repositions a node depending on the positions of its relatives. ecIO is a notification issued to a line's endpoint nodes to warn when the topology changes. The endpoint node has then the opportunity to reposition itself with AutoWeight.
  9. Restored the lwReturn locking mode which implements the skip-semaphore. MapScan might want to use this mode to avoid contention with another thread that have already claimed the allocator.
  10. Found the cause of a nasty problem that we've been chasing for some time, the spurious creation of orphan lines: TREE::FindCreateUML was turning line auto binding off, for some old reason. The TestThreads logic doesn't disable line auto-bind anymore. Generally speaking, in a multi-threaded environment, no thread can disable auto-bind without affecting the entire system. The flag must probably be taken out of the design.
  11. An attempt to turn TForm_TYPE_MemMap to refresh the memory map on its own blocking thread. Unfortunately, there are problems refreshing the bitmap that need fixing. Also, we must promote the drawing logic over to the core on a CANVAS stream.

Problems:

  1. (fixed) Expanding and collapsing certain proxies in the Scope View triggers the stream Open or Save logic that was primarily intended for the AutoLoad.BIN mounting point.
  2. (fixed) Unjustified 2501 or 1761 bytes of the thread's allocator during SelfTestMemory.
  3. (fixed) MemScan, SelfTestMemory and MemGrow are each wrong in establishing the start and end conditions for their iterators. They need consolidating on a common backbone. MemScan, for instance doesn't deal wit a list of objects at the very end of the heap.
  4. The auto-trace uses the following virtual methods vmtChange, vmtCreateInstance and vmtDescendsFrom, which must not be replaced by the DNA analyzer. These are exceptions to our unified VMT mapping which need to be revised and dealt with differently.
  5. Still having problems un-guarding the bottom of a stack that's larger than 1M. Using alloca(stackavail()) in GetStackLimit didn't help much.

A screenshot <here> might best describe these latest developments.


284 Wednesday, August 5, 2009

In this version we successfully construct the threads at the top of their own stacks. Each thread now initializes two nodes at the bottom of the stack, GOAL and CALL. The goal of the system is a place where the Inter Thread Procedure Calls (ITPC) are grouped and ordered, whereas CALL groups the thread's call frames intercepted by aouto-tracer. The solution to initializing the thread on its stack was to define a chunk of 512 bytes of amorphous space right at the top of the stack in TryStartForkThread which is the thread's startup, then promote its address through the core's CreateForkThread methods leading towards THREAD::INIT_THREAD. The buffer is said to be a thread's "precursor". The, in the thread constructor, a calculation is made to turn the difference between the original amorphous space and the actual space taken by the thread object as a small free block. The thread's name is allocated off this space. Thread construction proceeds with the self-allocation of GOAL and CALL nodes with their names at the bottom of the thread's space. The thread evolves by managing its calls, and eventually terminates by getting back to TryStartForkThread where the initial amorphous buffer, now formatted, would be destroyed by invoking the thread's dynamic destructor Free(). Grouping in GOAL and CALL has helped the GVF Enter routine to avoid the very dangerous real-time loops it did to locate the previous call frame and the free block of the stack's main expanse. The algorithm is more direct and faster now, should also be safer, although higher speeds often mean increased likelihood of running into errors of real-time contention. Thread's GetEndAddress has had to adapt to those changes. Thread destructor first deletes all sub-nodes in its scope, then the lines, etc. Thread constructor does not allocate a frame node as it can never be a safe morphic point.

Finally corrected the SelfTestMemory to walk the entire heap correctly from start to end, so thread's memory test was now enabled. There were so many problems with the loops, with inter-block's object list de-convolution, etc, reflecting an earlier state of core development. We've now merged SelfTestMemory into MemScan which does them both with a much simpler algorithm that traverses all the blocks. One problem with this version of the iterator is that by traversing left to right, it doesn't check the correctness of ordering the free and used blocks. Any errors in the this order would compromise MemReclaim and allocation in general, so the tester must be enhanced ASAP. The GUI map views filled by the new MemScan are now reliable, and the long standing errors and gaps the tester always reported onto the Morphic Core's top are gone. SelfTest on the morphic category still returns an error regarding the count of blocks (FIXME!).

The core's mmm_free and mmm_realloc memory management vectors were revised so that they locate the memory manager that allocated the block submitted for deletion or relocation, since it may now reside in thread stack space.

Enhanced the interlock to allow a thread to lock itself, such is the case with a thread that's doing memory allocation on its stack, and allowing a thread to lock onto another thread as if it were a common resource. The system still deadlocks the GUI easily so we experiment with a new SpinlockInterlock that processes GUI messages instead of going to sleep.

Enhanced NODE::GetName to return the name of it's type master whenever the node doesn't have its own name. The move is in support of call frames which are instances of a thread's uses relations.

(fixed) Most browse-able objects in the system are now stream descendents. The use of stream LoadContent and StoreContent on operations of expanding and collapsing tree nodes had to be disabled now. We need to rethink which nodes can and which cannot be stored to disk by this dirty un-mounting mechanism. A thread, for instance, may look like a stream with an open handle, but may not necessarily be loaded and stored this way. This logic was often causing system hangs and faults when trying to expand thread nodes.

Implemented a skip-locking solution to brace MapScan in TForm_TYPE_MemMap to avoid contention. Still no real solution though. Added a re-entry prevention for the threaded version of the scanner.


285 Friday, August 7, 2009

Fixed a series of problems found in the previous version:

  1. System shutdown was again failing with recursive destruction of CAT_MORPH. Found and fixed binding problems with the DISK nucleus. It has to be contained by the ABSTRACT nucleus that allocated it, not by the morphic cell. DISK must also be flagged as a disowned component. The constructor for DISK wrongly cleared the Disowned flag. Another problem was that during destruction, the abstract category prematurely detached itself from the surrounding morphic cell.
  2. Also responsible for various memory corruptions during threaded runs were mmm_free and mmm_realloc which needed to avoid using the top level CAT_MORPH type as their allocator.
  3. TYPE::GetStartAddress and GetEndAddresses were corrected further.
  4. The thread_precursor argument of CreateForkThread had to be modeled in UML, else the auto-trace crashed. The latest version of the tracer seems much more sensitive to errors of VMT layout.
  5. Constructors return null if the type master is relocating.
  6. The raw_Insert and raw_Remove aren't calling any proxy updates now, and the updates have moved to the locked Insert and Remove variants. Interlock is back on doing Insert and Remove with spinlocks.
  7. Updating the morphism path buffer cannot be done concurrently by multiple threads. Disabling.
  8. ThreadTest was now wrong in looping and deleting all children of the scope, which also include GOAL and CALL, and stores its nodes in a local nodes vector.
  9. TPrimaryExpression is a TStatement.
  10. The TypeInfo hierarchy of a thread cannot be logged in a tree view. Disabled the TypeInfo aspect bit from the mask of a tree-view's root.
  11. Added an address to the error message issued by a degenerated block.

Major enhancements and corrections

  1. The system can now defragment the main memory manager even when the auto-tracer is installed. It can also run the SelfTest thread with auto-trace enabled. The key here was to register in the thread's structures which object is being relocated and where, also which is its true handle. RelocationSource, RelocationDestination and RelocationHandle are new and the auto-trace logic uses those 3 new THREAD fields to resolve heap inconsistencies when the method being traced is also the relocating object. These fields are actually a SELECTOR.
  2. Added the AncestorVmt to DNA_ENTRY to replace the GetDirectAncestor method with a low-level DNA lookup. This new logic was required by the latest auto-trace which doesn't decode the "this", "type",. "method" anymore. There's a new and a better GVF auto-tracer, one that replaces the former expensive de-convolution logic of this-{Type-TIV}-method, with the much faster CALL analyzer followed by direct DNA lookups. TForm_TYPE_Aspect was enhanced to show the class of type and its direct ancestor. The super kernel binds the master OBJECT_BASE::DNAs into a tree structure
  3. A series of threading problems cleared away when we moved the thread's USE relationships onto the stack, thus preserving the state of the thread's environment.

GUI

  1. Enhanced Type View to be more useful with those stack frames. Clicking on the Overlaps or Gaps panels disable/enable the type's full map scan. With full map scan, the core calculates the total number of gaps and overlaps, and returns the calculated total object size. The effort in doing so is often too much, so the viewer flags those off by default. In "light" mode, only the 100 lines of viewable rows are updated. A change was required to TYPE::GetSize to have the format loop break-out early.
  2. Type View can now be called with a second argument to locate a type's instance in the list of instances and view its values. The format grid responds to drag&drop to loads the instance data into the type viewer.
  3. TypeView's instance list is now filled with the synthetic names of objects as brought by GetName. No-name objects are now harder to appear.
  4. Enhanced the type view by recognizing pointers to objects into the core's GetSize and showing their names. TypeView now shows the names of an object's parent/children and neighbors, which is more helpful.

286 Wednesday, August 12, 2009

Moved the HTML and SCHOOL classes out of the core and into their own categories of the OS. Added the SQL category with only one SELECT clause so far.
Reduced a few fields of the OTP category to somehow get closer to absorbing it into the core.
Added (disabled) the following fiends to STREE_BASE: Orbit, Magnitude, Flags and States.
A fix to the UML loader setting the SizeCache of fundamental types.


287 Friday, August 14, 2009

The major enhancement in this version is the analysis of a thread's inter-checkpoint gap, also called the uncertainty region. Between any two checkpoints created by the auto-tracer may be the stack frames of uncontrolled methods and free functions. A thread cannot be deactivated, persisted, exchanged, relocated, reactivated unless its entire stack is formatted. GenericVirtualFunctionEntry now analyzes and reduces the gap between the newest checkpoint and the previous one, creating call frames for any method that it recognizes in the morphology. Only methods that have pre-allocated a node header as their first local variable may be back-traced by the GVF gap analyzer. In the absence of minute format information regarding the local variables of methods, most of a call frame's extent remains amorphous constant data. GVF creates a local vector with the stack frames (ebp) between now and previous checkpoint. The vector is a mini-stack so that GVF wouldn't have to recurse. For each stack-frame, GVF decodes the caller's call instruction to see if a relative or absolute addressing mode was used, it calculates the address of the method invoked, then patches the return address to that of our GenericNormalFunction assembler shim which calls back GenericVirtualFunctionExit, with the proper stack layout, to delete the call frame and restore the return address to the caller. One unpleasant aspect of stack analysis is the parsing of all method instances to find each frame, which is a type n*(n-1)/2 algorithm in terms of effort. The only way to expedite the method location by address is to establish the order vector much like with our dictionaries.

TYPE::GetSize ignores USES relations regardless of its containment kind.
GetName for relations returns the supplier's name only if the relation is unnamed. There was a defect in browsing named relations.

Enabled the new ORBIT STREE_BASE fields into the kernel map.
Implemented the destructors and the relocators of categories HTML. SCHOOL and SQL so that the core can defragment and that it doesn't crash on shutdown.
UML loader is now posting an error if it finds virtual methods with no ENTER_FRAME statement.
Changed the Memory Map Viewer to show constant data region in white.

Carried out a cleanup and redesign pass with the following outcome:

  1. MORPH::Autoload stream is defunct. A thorough rework of the serializor is due anyway.
  2. Added the Video Camera Core category to the morphic core. We must somehow re-activate that logic. Need to watch TV.
  3. The top category doesn't have any fields now, so it's cleaner.
  4. The old and dormant POSIX category is gone. TSuperKernel is much simpler now. The user may descend from it to install more modules in Ring1, Ring2, etc.
  5. SERVICE is now defunct. Everything should symmetrically go through PORTAL.
  6. DICTIONARY doesn't own any fields now. All its flags have been distributed and aliased with others through the core. Its options were really pertaining to the SCHOOL class. All flags are now bit shift positions.
  7. CamCore doesn't sub-class the OTP mini-core, rather it descends directly from abstract types. Merciless reductions into this category.
  8. InitDxVideo, GetSample, and DoneDxVideo of DxVideo.DLL had to be stubbed-out for Windows7, else OS.DLL doesn't load.


288 Friday, August 14, 2009

A series of changes have to be archived after a long period of inactivity:

  1. AMORPHOUS::ContainerIndex is new. TYPE:;ContainmentVector in place of Unused1. Every object has this index into its allocator's ContainmentVector. This is to support dictionaries. DICTIONARY must provide a ContainmentVector so that tokens may live along with other objects in the same allocation, yet be indexed as grammar rules.
  2. SCHOOL now has and is a heap.
  3. Added TREE::GetBounds to get closer to clipping the 2D canvas.
  4. Added matrix::IsEmpty and matrix::Intersect to support clipping.
  5. NODE::GetBounds and LINE::GetBounds to aid with clipping. We're working on clipping the lines.
  6. FindCreate enhanced to create a null dictionary.
  7. All types are STREAM descendents, hence STREAM::GetShapeNames needs to return all the inherited node shapes.
  8. Reductions to the WebCam category just to get it running into K5.
  9. A fix to HTML::SetTypeMaster to properly bind the HTML category instance to its tag typemasters. The new SCHOOL works over the existing kernel HTML category without creating private ones.


289 Thursday, December 31, 2009

The last fixes for this year:

  1. The objective in this period was to recover as much of the application space as possible with focus on Edict and Gaius. Ran numerous compilation passes over the pool of legal text.
  2. The dictionary logic had fallen behind the defragmenting Morphic Memory Manager and needed updating.
  3. The notion of a constant is now more complicated since any heap is a constant of MaxSize.
  4. TYPE::GetSizeOfConstant is meant to be a common place where such calculations are made.
  5. TYPE::SelfTest cannot fail constants since dictionary is now a constant of size MaxSize.
  6. The new TYPE::ContainmentVector and AMORPHOUS::ContainerIndex implement a new TRAIT called Containment or 'C' gene which disambiguates memory containment from object polymorphism. MemRealloc updates the containment vector.
  7. Found a corruption condition in TYPE::MemGrow with relocating tokens of dictionaries when the UsedBlock is before the target FreeBlock. This means that rollover had occurred and block move must not be attempted. Found another problem in the end of MemGrow where the size of the final free block was wrongly checked.
  8. Fixed an indexing problem into the ContainmentVector with DICTIONARY tokens.
  9. The ENTER clauses which allocate the secret stack frame node may change the all too subtle register allocation scheme of the DECODE macro. Reverting to UNUSED entry frames.
  10. Dictionary now owns the word table so as to pass the integrity tests.
  11. Found a problem in DICTIONARY::ReEncode with resizing the null word.
  12. All tokens are now owned by the dictionary.
  13. Fixed the crashes in DICTIONARY::GetWordContainers.
  14. Increased the size of the encoding stream from 160MB to 180MB and the heam from 32 to 48MB.
  15. There's a defragmentation problem when only two free blocks are on the heap, their sum is larger than the request, but the algorithm must preserve somehow the content of the source.
  16. The largest token in a legal dictionary is <number>. It grows to over 4 milion occurrences in our legal archive. re-encoding needs an accommodating stack. The loader must be started with a stack of 32MB. A page fault was preventing us from forking with large stacks. Fixed the problem by reducing our morphic stack size by one page compared with that created by the system. The alloca call commits one more stack pages than we use.
  17. GetStackLimit doesn't call alloca anymore. The stack is committed at the beginning of INIT_THREAD with an indirect call to alloca between the current stack pointer and 4K above the bottom of the stack.
  18. Since the fields of DICTIONARY were reduced with those of TYPE, the only flags that now make sense are flDebug and flDisabled.
  19. SCHOOL is now protecting the DICTIONARY against a text that has no letters.


290 Tuesday, January 13, 2010

Pressing hard to consolidate the K5 kernel:

  1. This kernel version responds to MySQL login requests. We're going to be pushing to turn ANNA K5 into an SQL server.
  2. A change in the CAT_MORPH::DONE destructor. The system shutdown is very order-sensitive. Destroying the kernel's relations yielded a fault. Disabled this destruction until the destructor of RELATION takes them all down.
  3. We're now doing tight encoding tests of the legal dictionary. One scheme that seems to work is bitwise starting from 1 instead of 0 which currently fails in ReEncode.
  4. A change in STREAM::ReadBuf to distinguish between files and sockets by looking at the GetPos return value which is -1 for sockets. We must not clip the buffer length for sockets.
  5. HOST_OS::TransferLine was hanging when the stream data ended with 0 instead of CRLF.
  6. New button on MemMap to trigger a heap trim operation (lost code). Added some bitmaps for defragmentation.


291 Wednesday, January 27, 2010

Fixed a problem in OBJECT::Lock when locking a wire wanted to unlock the same wire and span for flags endlessly.

  1. Got the MySQL server to authenticate the user and send a phony database structure to the client.
  2. The meaning of the vaMembraneAffine flag has toggled to vaMembraneIndependent and all nodes are now born with membrane affinity.
  3. Changes in NODE::GetPoints so that when a membrane only ever has one child, it does not automatically center. This is so that the sub-node may move its parent when dragged.
  4. Got the center labels to work for free and cabled wires. Disabled the from and to label drawing.
  5. Allowing nodes to be wired even when their type dos not have an assignment operator.
  6. Enhanced the ScopeView New button so that it loads the currently selected node into the CurrentType proxy which is used to create new instances of various types. Null names build with the first letter of the type master plus a 4 letter phase.


292 Saturday, January 30, 2010

Started working on the graph editor and some graph related algorithms.

  1. Fixes to support the the new graph editor. FindCreateUML with separator path null....
  2. Another LINE option and flag to allow lines to draw their Information strings onto the schematic. This is to show the user-defined attributes such as cost, capacity, lower bound, and flux, that graphs use. Line labels are now rendered some 8 pixels above the line. Information fields render 8 pixels below the line.
  3. A new important mechanism was implemented: user defined attributes in clear text embedded in the information fields of objects noted as {name=value; } UserString is the low-level primitive to read, insert and update value lists into an object's information field.
  4. Added the new TREE editor to the MorphTypeLib. The editor is essentially a compiler/generator between the children of a tree and a grid that lists the attributes of those children. We may now edit all these details in a linear fashion, then compile the grid to apply the values over the tree. It's for quick editing of graphs. The editor can add and delete nodes and lines to a graph when the grid gets compiled. The editor features a graph generator from a list of tests.
  5. Allowing a non-locked object to lock another. Need it to trace graph algorithms in the schematic editor.
  6. Non-relation wires build their name string in the normal "from-to" order.
  7. Enhanced the TREE viewer with the Node-Node adjacency matrix viewer.
  8. Preventing the emission of 0 values as graph attributes.
  9. Added a series of graph structures to generate and a list of tests.
  10. Added the interlocking capability that shows a test result onto the schematic.
  11. Added the Run and Step actions to the test manager.
  12. Implemented FLEURY's Euler circuit test.
  13. Implemented Dijkstra's minimum cost node ordering algorithm.
  14. Implemented the simpler HasEulerCircuit, HasEulerPath and IsComplete graph functions.
  15. Added a flag field to the TREE editor. One used flag is R for Reverse the order in the graph children. Different order usually means a different result.
  16. Added an action to dump the graph structure in the debug log so that the node/arc lists may be cut&pasted into the source code.
  17. Independent trivial-reject tests for HasEulerCircuit, HasEulerPath, IsComplete so that IsHamiltonian may sometimes return true.
  18. Improved the BuildEulerPath logic.
  19. Added the the Labeling algorithms. Added the new NODE Label attribute.
  20. Added more test structures: "Square complete graph", "Dijkstra G1 min flow graph", "Dijkstra G2 min flow graph", "Euler graph <25,57>".
  21. Generating a test structure also fills the Graph and NodeNode incidence grids.
  22. Added a system of single-stepping through the algorithms.
  23. The new GetUnlockedPredecessors that labeling algorithm uses.
  24. A change to NODE::SerializeNode and to OBJECT::Lock in relation to the interlocking so that nodes locked by other nodes are drawn as locked onto the schematic.

TODO:

  1. Can and should create a graph just with the line grid editor, by creating the missing nodes.
  2. Need to interpret the path column and lock those in order as the grid is compiled. Must pass over the grid a second time to get the order.
  3. Need flags editor to select those compile-able columns.
  4. 0-int values should be deleted from the information fields to simplify the views.
  5. Need more tabs.
  6. The Node-Arc incidence matrix is view-only.
  7. The node-node incidence matrix is editable and compile-able.

293 Saturday, January 30, 2010

Pushing for a better tiling algorithm in the schematic editor.

  1. Added NODE::tsSpiralProgressive. Under this flag, tiling starts off with a larger value, currently set at 128, with which it distances the first level of nodes, and recurses with half the distance to tile the content of each node.
  2. Tiling algorithm also gets the spacing value as a parameter.
  3. A series of changes to the TileSpiral routine has helped eliminate the recursion.
  4. We're showing node labels only when their vertical size is above 8. We are representing tiny nodes as dots, and 16x16 shapes as bitmaps.
  5. Enhanced the Schematic Editor to set node sizes to {zero-label, 1, 8, 16} and to tile nodes {adjacently, equally spaced, progressively spaced}.


294 Saturday, January 30, 2010

A much better tiling algorithm ensuring the tightest spiral fit.
The NODE::TileSpiral algorithm:

  1. Checks each node against the environment rectangle and rejects those that don't fit.
  2. Doesn't take the inner rectangle as parameter. That info would be calculated upon the positioning of the first node.
  3. Doesn't take the vh and lr direction arguments, as the tiling function isn't changing direction by recursing. Direction variable is local to the function.
  4. The new level parameter isn't yet used.
  5. Tile doesn't move the category to 0.
  6. Heavily used types would only be over-sized to reflect their wiring only if they are being tiled in sizes greater than 16x16.
  7. Tile pre-tiles its children before spiral-tiling them. TileSpiral needs the (0,0) position of a node to mean that the node hasn't yet been tiled.
  8. TileSpiral now gracefully fills the 4 sides of a rectangle which is initially set to the bounds of the node-head of the list. It changes direction four times to circle about the head. It rejects nodes that would not fit into the environment box. It skips over nodes that have already been tiled. It recursively fills those square regions that occur at the corners and on the sides of those nodes being laid-out to increase the occupancy.

A fix to clipboard copy. Clipboard stream should not be locked into the creator thread. There would be a deadlock between the GUI and the clipboard maker threads.
Relations are now created as INSTANCE so as to show their fields in the schematic.
Cables are now being born closed and not filed.

  1. A much better IsInsidePoly algorithm based on the scan line intersecting the shape an odd number of times.
  2. rect::Includes and rect::Offset to help out with the tiling algorithm.
  3. Unsure still about how to interlock unlocked objects.
  4. Changed the node pick correlation to use rect::IsInsidePoly, which is much more accurate now.


295 Thursday, February 18, 2010

Moved the UserString from GUI Edit_TREE to OBJECT. Need those to set al kinds of field data, especially for debugging purposes.

Fighting against this TileSpiral algorithm... Bitterly hard... It's yet another multiply recursive algorithm, with a whole heap of direction changes along its loops.

Resolved with the tiling algorithm to produce compact arrangements.

Hiding the debug spiral drawing under the new ofShowSpiral NODE option. Spiral drawing sets and relies on user fields set inside the object information strings.


296 Thursday, February 25, 2010

Found the reason for the OtpDirView not logging the OTP tree. TTreeNodeExt should not have been disabled with LINK_TO_BASE.
Cleaned-up some duplicate symbols between PORTAL, OTP and WebCamCore.
ECHO is the new DBG macro for release builds.
STREAM::Truncate was failing through chsize. Now going to HOST_OS::FileTruncate.
UserString, UserInt and UserDouble had to be virtual so that loosely coupled core clients may access them without linking. NODE::GetNext and GetPrev had to become virtual for the same reasons. ABSTRACT::FindCreate is a new virtual router to LINE::FindCreate.
Added the new LST file type to Scope Viewer Open/Store and file drop. Added the LST file type to the ABSTRACT core. Its interpretation is of a constant open stream in the directory structure which can later be traversed by manipulator processes.
This version adds a compiler for AWB documents in text format.
The compiler is very much like the OCR on pixel maps in that it too crops according to preset regions where it expects to see the sender, recipient, transaction details, etc.
The AWB2XML convertor is a fairly complex piece, currently built as both a manipulator (.CMD) for ANNA's GUI or as a standalone executable. There's a build problem with BC502's linker in that the BASE statics are wrongly tagged for loader fixup. We so had to build the monolithic tool with CBuilder.


297 Tuesday,February 02, 2010

Added a barcode reader and OCR engine to the build structure.
The ocr_barcode.cpp module now contains all the code to detect barcodes on an input bitmap. The technique is to de-blank the image and retrieve a list of boxes with a high pixel density, then attempt barcode recognition within each box.
The algorithm returns a vector of entries each with text, code, and position.
It is the position we were so interested in, so we may triangulate our invoices and de-skew them prior to textual OCR.
The barcode recognition is weaker than that of DTK, but infinitely more valuable as it now permits an all-ours solution to the OCR problem.
We may now build monolithic versions of the OCR engine.
Some changes to DGRS were required to integrate the new barcode reader as a managed DLL. Some other fixes were needed in DGRS to convert foreign invoices into Proxima's canonical format.
Changes to the DGRS core so that the XSD.EXE tool may produce a schema file. All enums had to be prefixed with their containing class name so as to not clash when the XSD serializor flattens them. Certain fields that cannot be serialized cannot be echoed to a schema either. Using the [XmlIgnore] attribute.
Added the x64 target to DGRS. Barcode reader is legacy C. Succeeded to link it into DRS Core as a module rather than a DLL. A bridge managed entry point was needed to resolve mangling.
The 64Bit Barcode reader works!. TESS doesn't. Excel QWC10 cannot be loaded under 64Bit. The matter needs further investigation.
DTK doesn't load for x64.
Under x64 XSD produces a schema of DRS.DLL. The framework must be leveled at 64Bit or 32Bit for the tool to work.

Found and fixed an silly error in the stabilizor: ref_max_dy and max_dy were not set correctly.


298 Wednesday, March 10, 2010 (milestone)

Loading XSD schema files into ANNA K5's morphology is a milestone. The system can now load a schema and generate some native source-code from it.
LoadCMD is an external command that processes a stream node on K5's topology and creates the corresponding class hierarchy.
SCHOOL implements LoadHTML rather than LoadXML. The XML loader s now implemented as an external command.
Added TYPE::GetInstanceByName used by LoadXML.CMD.
Added the iftStream and ftStream for unknown file types that would simply mount stream node into the K5 active directory structure. Needed this for XSD and XML files.

Got ocr_barcode to also do text OCR. Got it to work in 64Bit as a /cli:pure object. Unfortunately its output is so bad compared to tess, it's almost useless.
Added the EEngineFlags enum to CAT_DRS to enable/disable the various engines that the recognition system builds with.
A change to allow the generic editor to open in the VS designer.


299 Thursday, March 11, 2010 (milestone)

Got the 64Bit DGRS to compile invoices. TESS is 64Bit now, linked in as a managed library.
The effects of this result is that the DGRS engine has become a large capacity device, capable of holding into memory the entire database, limited only to the available swap space.
OCR is some 30% slower in both barcode and text detection.
The issue of re-entering the text OCR engine is still open, the OCR cannot progress on multiple threads of execution. Another limitation of this 64Bit build is that ADO isn't working because unfortunately we cannot mix with 32Bit ActiveX controls. The consequence is that the 64Bit version cannot load the product lists from EXCEL. Luckily, the XMLs are interoperable between the 2 versions so one loads the product lists under 32Bit and saves the XML, then opens the XML in the 64Bit. Dropped the VB support. Dropped the ATL_COM module. There were subtle parameter passing problems with 32Bit ints.


300 Monday, March 22, 2010 (realtime-safe)

Another battle for real-time.

The outstanding issue was that ThreadSuspend may not complete in time for ThreadResume to commence, in which case the core would attempt to resume a thread which was not yet suspended, consequently doing nothing. The result is that suspend would eventually finish and there wouldn't be any way of resuming such a thread, since the core thinks it's already resumed. Here there's a deficiency of Windows, as it does not notify the user in any way that a given thread had indeed been suspended. We resorted to the NtQuerySystemInformation deep system probe to find whether or not a thread is really suspended before attempting to resume it. We'd be cycling about a resource's blocked threads to find one which is definitely suspended in order to resume it. The probe worked after some grief, however it unveiled what I think to be a bug of the 64Bit Windows kernel in it's 32Bit system emulation: the call returns an incomplete list of processes no matter how large the input buffer it receives. The last process to be created is always missing from the list. Anna's core spins about such an error so as to salvage its interlocking structure, and the only way to have it move forward is to spawn another program such as a notepad. The NtQuerySystemInformation is also hugely expensive as it serializes the entire kernel tree of processes and threads, when all that we're interested in is whether or not one of our own threads is suspended or not. The buffer is casually of 100KB in size so I'd expect that serializing so much info would be an effort of several million CPU cycles. The Core's interlocking scheme eventually worked with this probe.

Had to trace and correct some parallel access issues relating to the closure monitors of the interlocking system which at some point was wrongly reporting unbalance errors.
Added the OBJECT_BASE::State flags field where we isolated flags relating to the interlocking so as to minimize the suspicion that some code would inadvertently affect them. Access to those flags is through AtomicSetFlags and AtomicClearFlags assembler shims.

Realized that looping to insist that we set the flInLock bit in ThreadSuspend was very wrong, as it was only natural that as soon as this bit is asserted another thread to come through the Lock spinlock and set it again. This was actually the first main step towards correcting the interlock logic. Changed OBJECT::SpinlockInterlock accordingly.
Gave-up the idea of using the expensive IsSuspended call. OBJECT::Unlock is now checking the return status of Resume to jump past a thread that could not be resumed because it had probably not finished suspending. In response, OBJECT::Lock must spin-wait for the stInUnlock flag to clear so as to ensure that Unlock has finished re-arranging the interlock tree, and only then restore the resuming thread to the root of the interlocking tree. This was the solution to another haunting problem of the interlocking system.
Phantom proxy nodes were left in the GUI trees of Scope Viewers as the thread was attempting to destroy its call and goal sub-nodes off the main heap instead of deleting them from its private allocation. We've hunted this problem for many days.
Dumping the interlocking structure unprotected is too dangerous and we now avoid doing it in the main thread of our memory pressure tests that are not explicitly semaphored.

UPDATE_PROXIES is now private to Base.CPP.

LOCK and UNLOCK of STREE Insert/Remove are now concrete implementations rather than expecting macros, which helps debugging.

THREAD::RealID is a new field to record the global thread ID as the system creates the thread. We needed this field to locate our threads in HOST_OS_RT::IsSuspended in the list returned by NtQuerySystemInformation.

mmm_lock and mmm_unlock had to be called in MemRealloc regardless of re-entry.
OBJECT::SelfTest was wrongly issuing the "Object must be owned" error for threads which are essentially disowned (as for now). Had to relax the ownership test to only check objects that are flagged as being bound to others.

OBJECT::SetTypeMaster constructor now serializes access to the critical region of destroying the proxies.

Setting the VMT of an instance to that of OBJECT before calling SetTypeMaster was wrong, as it confused the IsObject method in certain real-time scenarios. We're now leaving the VMT as configured by the caller, and call SetTypeMaster statically.

Re-ordered some of the trivial tests of STREE::GetCommonParent to happen earlier so that the error message printer that catches an unreasonably deep interlocking tree does not get abused.

Serializing access to the critical code of adding/removing VCL tree nodes. try/catch around de-references in TIsNodeClient::GetServer to have it survive better the aggression of the core's real-time processes.

Corrections to the selection dialer so that it doesn't crash.
No more red background for selected nodes.
A fix to getting the name of an unnamed autotype.
A change to GetPoints so that the content of a relation's supplier aren't repositioned.
The uKernel is not selectable in the schematic editor, it cannot be locked.
A fix to OBJECT::Unlock to allow objects to unlock another object.
The canvas of the schematic editor is now child of uKernel.


301 Friday, March 26, 2010 (biological system)

An effort to class ANNA K5 as a biological system. A demonstration with this system was carried-out at The University of Bucharest, School of Mathematics and Informatics with those results, during a lecture on Syntactical Modeling of Biological Systems.

The following changes were necessary for the demonstration:
Some corrections to the SetTypeMaster default constructor/destructor and the serializor of categories ABSTRACT, HTML, SQL, CamCore, SCHOOL, and MORPH. The aim was to be able to construct another instance of the morphic cell.
The SQL category was extended with all known MSSQL keywords.
Two more static NODE options were added, one to control the display of Glyphs, our molecular representation analogous to Source Homology 2.
Threads are currently banned from serializing to binary streams.
Preventing the serializor from moving up to serialize the uKernel.


302 Friday, April 09, 2010 (producer/consumer)

This version has passed the real-time tests of compiling the entire Romanian Legal Archive in 8 Bit link-size with a bi-threaded consumer-producer model.

  1. Regeneration from the model.
  2. Added the "orphan" argument to TREE::Remove, so that destroying a node doesn't go through inserting into the K5NULL constant before the final removal. That intermediate step was too noisy and dangerous.
  3. Another synchronization mechanism: semaphored AddChild/GetChild in trees to implement the producer/consumer model. The new mechanism is currently employed by SCHOOl to load and pre-process HTML documents on a thread that is different to that which winds text up onto the archive. The number of slots is set as the magnitude of the RESULTS node of the SCHOOL. The producer uses AddChild which blocks when the queue is full, the consumer uses GetChild which blocks when the queue is empty. Furthermore the producer AddChild resumes the consumer threads if such threads are found to be blocked onto the common tree node while the consumer GetChild resumes any producer threads if such are found to be blocked onto the common resource. The difference in thread's quality as being a producer or a consumer is marked with 2 and 1 in the interlock information field of the common resource. The producer/consumer logic was fully tested with one producer and one consumer during a dictionary encoding session for the Romanian Legal Archive. Scenarios of multiple producers and multiple consumers have not yet been developed.
  4. Added TYPE::EmitSchema. Not yet functional.
  5. Found a problem in the memory test when the allocator is an auto-type (morphic pivot) but it is not self-contained. Such an instance is called an auto-instance and it is counted separately from other blocks so that the equilibrium of counters is preserved. Without this change any DICTIONARY failed the memory test.
  6. SCHOOL::Options are now accessible through ABSTRACT::GetOptions.
  7. OBJECT_BASE::Flags and States are not volatile anymore, as their storage class was never the issue in the interlocking mechanism.
  8. Dictionary encoding was failing because of some childish problems in HTML::ResetAbnormalSchema and ClearAllTags. Indexing was wrong. The relation aspect instead of the type aspect was tested for tag flags. Immense pain to get the dictionary to encode again correctly to diagnose this source of encoding failures. Enhanced ClearAllTags with a boolean to help keep the html scope during the compilation of legal text.
  9. Added the TIsNodeClient::ofChangeNotifications option to globally disable the noisy VLock, VUnlock and Change proxy notifications. The Scope Viewer is much friendlier now and still shows the interlocking mechanism in action.
  10. Added the new GUI EditSchool to allow us to set the parameters of a dictionary encoding process.


303 Thursday, April 29, 2010 (Database Storage Engine)

Nearly 3 weeks of work to implement a bridge between MySQL v5.1.42 and ANNA K5 working as a storage engine. ANNA Version 14.303 is a storage engine for MySQL. DGRS has a dual persistence path to MySQL and MsSQL. Foreign intrusion threads are now wrapped and managed. The TYPE::UpdateInstance exchanger is operational. Echo-relocation is now working. Enterprise GUI Frame may be launched from the core. ANNA OS may now run as a service.

Modifications to ANNA K5's METACORE to function as a storage engine:

  1. BASE::ObjectClient is a stub OBJECT_BASE wrapper used to crate proxies to core objects. There were unwanted recursions on methods such as set_Name in the former use of OBJECT_BASE.
  2. BASE::ThreadWrapper is a new class used in every API to wrap around foreign threads and turn them into managed K5 threads. All threads that traverse ANNA's structures must be managed so as to be able to interlock with resources.
  3. Of great importance to managing threads is to know the extent of their stack. Added RawGetStackBase and RawGetStackLimit() to read the stack limits from the ES segment directly.
  4. Added a new GetThreadStackSize method to HOST_OS which retrieves the stack limits from the TEB of Windows.
  5. Managed threads must destroy their heap even when exceptions occur.
  6. The new proxy and the new managed thread are now used by the ANNA_DB_BRIDGE API DLL which in turn is now used by the MySQL ANNA_DB storage engine.
  7. Added several new fundamental types in the abstract core: Int64, Float, Double, Time, DateTime. Gone to version 14th of the file system.
  8. Enhanced the constant node to stream conversion to 64 bits to see the masks of the constants of the new data types.
  9. Enhanced TYPE with some database specific logic: TYPE::UpdateInstances is a new method to marshal type instances in order to read/write/replace/delete its field data. This method copies data between kernel space and user space and it is an important new development in the Core. TYPE::Select is analogous to a stream seek and uses the type's interlock to lock onto the n'th instance. This method is now used by the ANNA_DB_BRIDGE API to scan-select rows in tables.
  10. Class ANNA_DB is new in ANNA K5's SQL category.
  11. Enhanced the TYPE::GetSize logic with the SMapFormat::range limits so that the call-back may trigger only within the range. The new TYPE::UpdateInstances is the first example of an update done on a configured range of columns only.
  12. Enhanced/corrected the ECHO-LOCATION logic of TYPE::DoRelocation to use partial type map scanning and only relocate the user region of a morphology. Relocating database table rows was a major struggle that could only be resolved using the echo-location algorithm. Since it involves map-scans it tends to be much slower than the Relocated methods. The memory defragmentation consequently uses the embedded Relocated and RelocatedString methods for the system portion of an object, and uses echo-location for the remaining portion.
  13. Enhanced the echo-relocator to cope with USES relations.
  14. CAT_ABSTRACT::CTR_RELATION is the virtual constructor for relations for use in de-coupled client programs.
  15. Found and fixed several crash conditions with an increasingly aggressive real-time metacore.

Modifications to get the GUI to attach to the core on demand.

  1. The Enterprise frame is now a DLL with the entry-point in CreateTray.
  2. Any thread may now create an Enterprise frame even in the absence of an Application singleton so long as it calls the GuiPeekDispatch in a loop. GuiPeekDispatch was enhanced to do message translation, so as to process keyboard strokes as well.
  3. AnnaK5UI.DLL is new in the CBuilder Navigator group and builds the DLL version of the Enterprise GUI Frame.
  4. We were getting all kinds of exceptions during frame creation due to the printer spooler. Deleted the print dialog built into the Scope View, and creating it on demand.

Development of the ANNA_DB storage engine for MySQL and the ANNA/SQL/dbBridge.DLL module:

  1. AnnaDbApi.DLL is a flat C API to create/delete/rename/alter tables and to seek/read/write/update/delete table rows. The module is built with Borland tools and requires a conversion to a Visual Studio - usable import library. The command to do that is: "SQL>lib /DEF:AnnaDbApi.def /MACHINE:X86" in the Windows SDK 32Bit build environment CMD.
  2. Copy the AnnaDbApi.LIB over to MySQL's example storage engine directory along with the annaDb.H and annaDb.CPP source files for the engine with a command such as "copy AnnaDbApi.lib X:\Sources.DEV\mysql-5.1.42\storage\example\" and have the AnnaDbApi.DLL module available on the path of the MySQL service.
  3. AnnaDb.H/CPP is an instance of the example storage engine. The main area of difficulty in this module is to convolve/de-convolve the record buffer for a transaction and build a neutral representation that can be passed to ANNA for marshalling.
  4. The ANNA database engine implements all the fundamental MySQL types and the bridge aliases them to ANNA's own data types in the ABSTRACT category through USES relations. Anna's internal Type Viewer would be able to operate on the tables using its own interpretation of the column types.
  5. Our storage engine does not implement any keys at the present time. Our database client uses a construct we call BOND to correlate objects.
  6. The bridge turns the external threads into internally managed threads for as long as they stay in the cytoplasm of ANNA. This is "controlled invasion". Once the client is in, its thread would be treated as any of the inbound threads of ANNA, participating in interlocking of threads and resources until it retracts from the cell.
  7. The AnaDbApi.DLL bridge filters the requests to create tables and opens-up the Enterprise Frame whenever a CREATE TABLE anna_k5_gui (dummy INT) query is issued.
  8. The new system of launching the GUI on demand is useful when loading ANNA as a service.

Modified the Document Generation and Recognition System to work with both MsSQL and MySQL:

  1. Changed the database connector.
  2. Cloned-off the PersistenceService CSharp modules into MsSql and MySql namespaces.
  3. Converted the MsSql specific code of the PS to MySql.
  4. Changes to the data types. The order of emitting the fields is natural for MySql. No brackets around user-defined names for MySql.
  5. Each persistent field of DGRS is twice extended with attributes for both MsSQL and MySQL.
  6. DGRS now has the ability to store to either one of the MsSQL and MySQL database servers. The choice is made via the file save as dialogue where a connection string may be given instead of a file name. If the file name ends with the .sql suffix then the MsSql persistences driver will be used. If the connection string has the .mysql suffix, then the MySQL persistence driver will be involved. The connection string must look like a possible file name in an existing file directory so $ is used instead of backslash.

An attempt to rebuild ANNA using the Visual Studio 2008 tool-chain.


303 Thursday, May 06, 2010

The BASE::CoreApi is a new source-code module to hold ANNA's METACORE_API as the converse of the HOST_OS API. Is a better place for the database bridge, only that it affects the GUI startup. We kept the AnnaDbApi.DLL module for MySQL to avoid the startup ordering problem.

METACORE_API is the converse of HOST_API, both should be seen as part of one bi-directional API.
CAT_ABSTRACT::SetTypeMaster had to set the DNA for the new fundamental types Int64, Float, Double, Time, DateTime.
PORTAL is a memory manager with internal free space as configured by the database API.
PORTAL and CAT_ABSTRACT define some dummy profiles now.
TYPE has gained several new profile counters to do with databases.
The UPDATE_COUNTER macro works on (LastEffort, TotalEffort, Count) profile triplets.


304 Friday, May 07, 2010

TYPE::MemRelocate and TYPE::Activate are new functions to use in relocating a sub-allocation space.
A change in OBJECT::SetTypeMaster to preserve the Handle as set during default construction.
I'm struggling hard against the relocation mechanism to get it to relocate entire sub-allocations within the topmost memory manager. MemGrow is now capable of relocating a sub-allocator as far as to fix-up its intrinsic relocations, but is still broken in that the echo-location doesn't work completely. MemGrow uses TYPE::SerializeHeap twice during relocation, once to inactivate the sub-heap, another to re-activate the heap at its new location.
TYPE::SerializeHeap has changed, and probably doesn't work anymore as intended during startup BSS load and shutdown BSS store. SerializeHeap does not faithfully implement the latest memory scan logic with all its exceptions, so it would only persist a heap that doesn't have intricate objects.
The pointer protection mechanism had to be reworked. The meaning of RangeStart and RangeEnd was all wrong, and has been corrected to now mean the start and the end of a protected area outside of which pointers are considered to be valid. Two new fields were added to the SProtectionInfo structure to hold the limits of reference relocation. RelocationRangeStart and End are now the limits where the protect and unprotect routines triggered by GetSize(mf) are allowed to act. We need the ability to set those limits to partly activate a range of memory while a larger encompassing range is protected. The relocation logic of MemGrow would promote an entire heap to an amorphous block but need to keep its header unprotected.
There's a new TYPE::MemRelocate method with the basic heap iterator built-in that can take a command parameter to carry-out a specified function. This is a scrap-board where we currently test some aspects of echo-location.
This system initializes with line auto-bind enabled at all times. A DNA reordering was required so that RELATION is constructed after LINE. A change in the ordering of events in CreateInstance has helped do away with over-locking the memory manager. The core is consequently constructing with no closure errors. This is therefore a cleaner meta-core.


305 Sunday, May 09, 2010

Fought hard to relocate a dictionary.
NODE::Active was wrong in relocating the handle of a constant in the opposite direction.
Found that the concept of ContainmentVector and ContainerIndex as an order different to that of TypeInfo.Vector and TypeInfo.Index was insufficiently thought through. Dropped the concept and went back to a dictionary that holds its index as its DNA.
Dictionaries are constants implementing the TypeInfo.Vector and shold therefore not issue the "Old style constant binding" integrity error.
The index of the dictionary is now constructed within the dictionary so as to follow it around. The same could be done for the archive band.
These dictionaries are now relocatable. They are not yet resizable, though.
Some important logic was missing from the TYPE::MemGrow relocator to do with updating the TIV after object relocations. The dictionary handle was also wrongly (de)convolved during SerializeHeap. SerializeHeap also had to update the TIV. SerializeHeap invoked over a dictionary had to protect the head but also marshal the heap and not hop over it. Dictionaries are therefore an exception to the way SerializeHeap iterates objects.
CbMapFormatRelocated was enhanced/corrected so that on the second phase of relocation (reverse echo-location) to also bind the TypeInfo.Vector back to its owner.
Dictionary relocation must be protected. A dictionary must be locked with TLock before relocation and unlocked afterwards. This should be done for all objects in the heap but a deadlock is currently apparent with I don't know what member. An object cannot be relocated unless its users have been blocked. If users currently block the resource, the heap defragmentation has to wait for the rest of the objects potentially deadlocking the allocation. Defragment could be made re-entrant to avoid this type of a scenario.
Implemented thread termination via a thread->Free call that sets the new THREAD::flCancelled flag and then spins the caller until the thread has gracefully exited. To stop a thread one simply deletes it. A new destructor DONE0 was required in the OBJECT_BASE and a corresponding override in OBJECT to bridge to the signed destructors. Tthe former logic of THREAD::Free moved to THREAD::DONE.
The Schematic Editor locks the canvas just before serializing and unlocks it immediately after so that the canvas is mostly unlocked and relocation operations don't block. The SE unlocks the canvas without recursing so that all the assets of the canvas (ie the selection interlock) is preserved.
The new relocator locks the object before moving it.
Removed the construction of spurious node proxies for threads in the Scope Viewer.

The bulk of the logic of TYPE::MemGrow when to do with relocating objects was moved into the new OBJECT::Reallocate virtual method that moves an object to a new address. It does so by protecting the object and then promoting it to an amorphous block, then copies it, demotes it back to the original type and un-protects it. It finally relocates and echo-locates its references. Fixing-up the STREE chains is required on both phases of DoRelocation.


306 Friday, May 15, 2010

Added an enormous amount of tracing in the core's interlocking logic to catch the remaining real-time defects. Our Lock/Unlock pair was behaving heretically compromising everything.

The most significant improvement in this version of is the changes in OBJECT_BASE::Fork that now allow it to be re-entered. Fork was a major source of deadlocks in the system and the reason for a whole scheme of protections which have now been relaxed. Threads can no be spin-locking from the Core's point of view, yet execute GUI message pumping. Threads may now be forked while another fork is in progress. The key to doing this is to use Fork's stack frame as the spinlock variable. The frame uniquely identifies the (caller, object, child) triplet so that no shared context is required. Fork has now an extra wait argument, normally invisible to the user, used within Fork to spin and reset by DoStartForkThread as soon as the child copied the stack frame from the parent.

Thread forking is an example of a partial DNA replication.

OBJECT_BASE::CurrentThread returns non-null only if the current thread is managed.

ThreadWrapper destroys itself by calling DONE0() onto the thread object.

Changed OBJECT::Lock and Unlock yet again. Unlock comes on one thread to resume another. It doesn't remove the other thread from the object's interlock. Instead, the resuming thread will fall through the continuation of the Lock code which suspended it and remove itself from the interlock. The Unlock code waits for the resuming thread to have removed itself from the object's interlock. A thread is removing from an object's interlock only as a result of an Unlock on that object, so it must wait for the Unlock to exit before exiting the Lock code. There has been a tremendous amount of effort going into correcting this Lock/Unlock logic.

Added another closure monitor to do with the equilibrium of STREE_BASE::lock_count which should never be greater than 1. In other words, elementary operations on BOND structures must not be re-entrant. STREE_BASE::Root was re-worked to avoid the recursion and comply with this requirement. Added closure monitors for the new enqueuing counters of AddChild/GetChild.

Added the stVerifyLU OBJECT::State to be used as a measure of the lock state of an object complementary to and simpler than the Interlock.Parent. The locking methods were setting this and testing it to hunt for a defect in the logic of insertion that cleared the parent pointer compromising everything. The faults in this kind of logic are end-game. Tracing the context of such an error required us to build history checkpoint buffers with the paths taken through Lock and Unlock on a per object basis.

OBJECT::TLock and TUnlock were renamed to simply Lock and Unlock since they are not virtual and don't clash with the parametered Lock/Unlock during TIV mapping. Added OBJECT::TryLock and IsLocked. Added 3 new closure counters enqueues, dequeues and interqueues to measure AddChild and RemoveChild closure.

Enhanced the OBJECT::SpinLockInterlock with the new waitFlags where the user may specify the lwReturn value to designate a TryLock request.

SCHOOL editor has a combo-box with the paths of all the test and legal text so that we know where they are in the dev environment as I keep losing them. The SCHOOL editor also launches the producer thread from a new launch button, so it would not be the responsibility of the Core to filter the file extensions to know to run the school. The school can therefore load into dictionaries files with any extension that possibly overlaps the range used by the core's importer. Deleting the school's producer also cancels its consumer. Producer finds the leaf during a locked section. The producer thread runs the HTML compiler excluding any other producer so that tag->get_NextInstance doesn't bleed over to another document. The consumer thread locks/unlocks onto the dictionary.

Enhanced the DICTIONARY with the GetWordNameW and SearchStringW new methods capable of transferring wide character set strings to and from the dictionary. These are used to map and un-map the Romanian ÂâĂăÎŢţ diacritics. Dictionary TDSpinLock is now an indirection to its Lock/Unlock methods, so a dictionary is now a true mutex. As the dictionary was re-generated from UML, all UNTESTED have now turned back to ENTER_FRAME, potentially affecting the stack layout and also the register allocation. A full dictionary encoding must be carried-out again.

MemRealloc has to update the TypeInfo.Vector indexes during relocation of tokens in a DICTIONARY. Updating the MemRealloc profiles.

THREAD test suite enhanced with the Chinese Philosophers test. This test is quite important in stressing the OBJECT::TryLock method. The philosopher eats only if he can pick both forks from his neighbors. The table gets wider as these people eat. 'The test is a visual representation of the fairness of ANNA's dispatcher which is track-able onto the schematic editor. The original shape of the round table must roughly be preserved while the test progresses. The test is also important in that the test threads have interlinks and can be manipulated in the schematic as any other nodes. A lot more care was needed here to get the mater test thread to construct the vector of forks while the philosophers are kept at the entry barrier and delete those forks before releasing the exit barrier. The real-time test suite can now be terminated by deleting the threads involved. The editor for the ABSTRACT category has a new checkbox to enable/disable the integrity tests during the run of our real-time test suite. The issue here is that the Integrity Test cannot lock onto the objects that it tests, so it could crash if the structure changes while the thread is running.

THREAD has gained an Options field with the following options defined: ofEnsureSuspension to enable or not the IsThreadSuspended system call, ofTraceDispatches to pollute or not the log with interlocking squiggles, ofInterlockingNames to enhance or not the object name reporting with locking status. The editor for the ABSTRACT category has new checkboxes to set those options.

There was a problem with our internal THREAD::ID which should not have used the RunningThreads counter which decreases upon thread destructions this leading to duplicated Id's. Added the new THREAD::ThreadsEverCreated counter to build our IDs from.

Added a diagram with the system's real-time objects.

ABSTRACT editor has new checkboxes to set the new options of the core. ABSTRACT has the new ofNoProxyUpdates option that can be set from the editor. It disables GUI updates at the earliest place. This flag was used during the exhausting debugging of the core's real-time problems when we (wrongly) kept suspecting the GUI proxies to be the culprit.

Re-inserted the disk into the DrawingBoard, and the board as a direct child of ANNA. DISK is a size-inductive auto-relation that needs to be inserted carefully.

launch_gui is now exported. We're trying to get the Loader.EXE working.

With some restrictions now gone, all spinlock loops can now be processing GUI events.

Schematic Editor does its redraw by forking a thread separate from the GUI main thread. The canvas is generally unlocked. Preventing re-entry in the Schematic Editor's MouseMove handle.

Enhanced Gaius to use the new GetWordNameW and SearchStringW wide character text methods of the dictionary.

A lot more work went into making Gaius real-time safe with dictionary TryLock/Unlock statements around the refreshes of the lists, etc.

Added MorphTypeLibMono.TED to the navigator's group.

TIsNodeClient can now be expanded or collapsed even when it has no children.


307 Wednesday, May 19, 2010

This version of ANNA has passed the IR (Information Retrieval) and IC (Concurrency Implementation) tests.

We had a problem with the producer locking the HTML compiler where the lock did not hold. We verified that the lock was placed using a global semaphore variable which we verify to trap when html->Lock bleeds. We now pass such values as a hint argument to the lock method so we may trap there and check the interlocking context.

Found a synchronization flaw in Lock after the thread resumes and claims the resource too soon, before the unlocking thread had a chance to remove the object from the interlock. This lead to invalid interlocking order and to inter-thread bleeding such as with the HTML lock.

There was another flaw in GetCommonParent affecting the Lock logic in that the method wasn't protected with the global insertion spinlock DoInsertRemoveLock so the tree may have changed during the climb to the root.

We've brought back the manual interlocking for the schematic editor by ignoring the timeout value and locking solidly instead.

Made OBJECT::TryLock to return faster by checking the object states before entering Lock. We thus avoid sending a thread to fight for an object's spinlock.

There was a flaw in TYPE::Reclaim where an object promoting to AMORPHOUS between two used/free blocks was wrongly setting the MaxFreeBlock indication (rover) to point to a used block. This lead to complete anarchy in the memory manager. A swag of new tests surround the issue to catch such problems.

There was another error in TYPE::MemResetCache where loking the heap did not encompass the taking of the first block. Memory conditions would almost certainly have changed in concurrence with other threads by the time the loop was started. The method is used by those who allocate large blocks to bring the free-block indicator back to the start and thoroughly search the entire heap, thus avoiding defragmentation operations.

We've gone back to locking the memory manager during the entire CreateInstance call. Reclaim locks the memory manager over its entire operation.

Larger heap to 512MB to hold two dictionaries at once. Dictionary Initialize has gained an argument to set the size of the encoding stream. School sets it now to 100MB.

Core loads the MorphTypeLibMono.TED library of editors.

Another CAT_ABSTRACT option: ofIntegrityTest. The issue here is that the self test cannot be launched in every running context as it does not lock any of the structures that it tests. Objects may change states or they could completely disappear during the test which could crash the test. Disabling all debug self tests with this option.

The SCHOOL constructs two dictionaries now one with and one without diacritics. It remembers in its flags which dictionaries it initialized and what queues so as to avoid locating by name in the shape-shifting tree. The same reason for inserting the archier thread in the scope of the dictionary it manages. Consumer/Producer is the default model for the school now.

Added TrayMainTiny to the build. It has the simplest logic and the lowest linkage.

Schematic Editor forks a separate thread to handle MouseMove. It does so to allow it to block while the SE can still re-enter its message processing loop.

Refreshed the UML to match the current core.

Locking the memory manager in CreateInstance lead to closure errors at system startup. Not locking.

Enhanced CheckClosures with a parameter to extract the dispatcher state structure from the core and show it in the ABSTRACT editor.

Severed the hard links that have formed in time between the GUI to the core.

This version has lost the persistence services for MySQL as the GUI fails to load.


308 Thursday, May 20, 2010

Got the persistence services working again.

Repaired the GUI to start-off differently. We now have a tiny LOADER.EXE windows program built with the lowest BC502 tools which calls 3 methods of the METACORE_API: init, LaunchGUI, and done. These functions load the BASE.DLL, load the OS.DLL, LaunchGUI spawns a secondary thread on which it loads the MorphTypeLibMono.TED editors family, creates the tray application and loops for messages until the GUI exits, then unmounts the TED. METACORE_API_done waits for all the threads to have exited the kerenel before unloading the OS. The leap forward here is that one only needs to link to BASE.DLL to indirectly gain both morphology and user interface. We're somewhat closer to the model in which we want to run the software off Bianca with a loader that extracts the other modules off the raw partition.

DxVideo.DLL is obsolete and its logic moved inside BASE. AnnaDbApi.DLL is obsolete altogether.

OS.DLL doesn't mount the GUI TEDs anymore. That was a circular dependency leading to trouble in our initialization of the GUI on secondary threads.

WebCamCore TED was merged into the core's TED to build one single GUI.

Revisited the core destruction issues. Out metamorphic cell destructs itself by apoptosis. The order of destruction has to differ from that of construction. The system DNA is parsed in the reversed order with an exceptional inversion at the LINE - RELATION pair where lines must be destroyed before relations.

The scheme of nested locks on memory managers was inducing closure errors upon the initialization of the core. Went back to sequenced locks-unlocks. We need to manage the main (creator) thread as well. We must therefore postpone its attaching to BASE.DLL all the way to the initialization of the wrapper.

An error with a thread performing StackCommit is acceptable only if the thread is that which gives birth to the CORE and comes from a Microsoft program which has already committed the stack.

DxD and DxVideo are now built into BASE.DLL. DxD locates the Direct3DRMCreate function in D3DRM.DLL at runtime, thus avoiding a hard dependency on an old Direct3D interface.

Found and fixed the reason for failures in TED when built with static RTL and VCL: HInstance must be initialized! Windows RegisterClass fails otherwise with error 1410 meaning "class already registered".

ANNA's entire GUI is now monolithic. The model now is Loader.EXE+Base.DLL+Os.DLL+MorphTypeLibMono.TED+{*.CMD}.

Found and fixed a nasty problem with this build: failures in Borland's memory manager due to IsMultiThread not having been set to true. As a consequence, the memory manager failed on tree view and chart related operations. Resolved.

Split the legal corpus onto the 1997-1999 and 2000-2010 ranges.

Enhanced the SCHOOL to construct the 2 dictionaries of different, create 3 folders Entry, Queue and Exit for the workflow, spawn 2 archive threads to handle the consumption of documents. Implemented the diacritics IsRO sensor as part of the archive thread workflow. There are only 3000 or so texts with diacritics amongst the 50,000 in the corpus. InitEnvioronment creates once and then locates each time the school's repository.

Found (again) a performance degradation in our memory manager where the heap's revolving free block pointer was not set after a block split and the lookout for free space always commenced from the beginning through all the used blocks. MaxFreeBlock is now set to the TreeBlock remaining after each block split. A refinement was possible but not chosen: updating the MaxFreeBlock pointer only if the new ree block is larger in size than the previous one. However, in doing so we would have lost the tracking capabilities of our memory viewers.

We crash if we don't protect DONE/Reclaim with mutual exclusion in OBJECT::Free. The lock/unlock is back, so would be the closure errors during core initialization.

Built the tiniest possible loader.EXE with no startups from Borland.

Effort to build an all monolithic version of ANNA. Linking in Morph.lib faults Borland's linker no matter what we try. It does not seem to be to do with debug symbols.

Found the reason for the failure of Borland's linker: vvmt.asm defines the INITDATA, and EXITDATA to take the labels of the data segment. These constructs crash the linker. vvmt must be the very first module to link for such references to work. Avoiding by un-defining the code. An all-out monolithic build was produced (again).

In the startup of the monolith we need to reference all the editors so that they get extracted from the library. Core must initialize before TED which is why the 99 start-order of the main as opposed to 100 that of the TED.

Resolved the problem of not finding the *.DFM files of WebCamTypeLib. The path to those files must be included in the monolith's search path. The linker looks for them there since they are not built into the K5Ui.LIB.


309 Wednesday, June 02, 2010 (DNA-based declarations)

A major new enhancement in ANNA's core: DNA-based declarations. Those are best described in the DNA Declarations document. In "short" however, a DNA is a multiple connector whose individual connectors are relations to and from types we call Genes. The relations themselves are of two types to give them distinctive roles in aggregating the genes. One such complex declaration looks like the following 3-stranded hairpin:

have WorkersAndCompany: Worker *T2*AC32A16 null V char T16G32vT25 null V Company A25T7*A2 null;

The above declaration creates a company structure with 7 workers contained by value, each worker structure has a pointer back to the company, a 16 char job description and a 32 char name, the company also has a 25 character name.
The compiler constructs metamorphic complexes for single stranded DNA, double stranded DNA, partly-double stranded DNA with sticky ends and sticky middle portions. The compiler also creates non-palindromic enzymes of the likes of Eco-FocI.
All DNAs are thread structures with the x-over of genes allocated internally at the bottom of their stack. The threads are born "parked" in their initial container and generally asleep, but ready to start acting on eachother.
This is the beginning of ANNA's DNA-based language, designer and simulator.

There's a new relation flag flXREF to create relations with an XREF typed wire. We're now building the Adenine-Thymine bond using a normal relation and a Cytosine-Guanine bond using XREF relations to distinguish the two.
Enhanced LawView Gaius Corellator so that it runs the DNA new compiler when the esfMRF flag is set.
A fix to the memory manager to allow us to delete an object from a saturated allocator.
There's a change to TYPE::GetSizeOfConstant to drive the same logic of node, dictionary and portal now also through TYPE.
Re-enabled and enhanced the logic of flScalar types. The core prevents such types from growing and their output relationships from inflating. We needed flScalar types to support DNA-based declarations which could create containment relations pointing towards a void type that must keep its 0 size.


310 Saturday, June 19, 2010 (Physiology)

A major new enhancement in ANNA's core: the Physiology Analyzer. The matter is fully described in the Physiology RO whitepaper. I'll only be able to afford the short story here and with accent on the technical aspects.

Physiology in any system is everything to do with state transitions where "state" is the current value of a variable. Threads are the artificial entities that represent a user within a computer system. Threads act enzymatically on a system's topology ever changing it and ever changing themselves. Threads are the enzymes of ANNA K5. Threads compete for resources in multiple ways, the lowest of which is best described by the system's Interlocking. Watching the ever changing interlock tree is too difficult so a projective endomorphism of the system's core over a steady plane was necessary. The Physiology node is that plane of projection where threads and resources are specialized and replicated with the formation of a persistent graph which is a trace of the volatile interlocking relationships. Every thread uses a set of resources during its lifetime of which one is remarkable as it is the most recently used one. The totality of pairs (thread, last-resource) is a system's multi-state. It is this multi-state that the Physiology Analyzer has set about identifying on a schematic and a grid-like view.

The story was then generalized to say that the Interlocking is just one special tracing algorithm in the system that unites clients to resources, and that other algorithms are possible such as those described by a set of checkpoints that the developer may scatter through the code under a common umbrella. For demonstration purposes and not only, we've placed such checkpoints in the SCHOOL subsystem to trace the rather complex workflow of the formation of an EDICT5 archive. Two new concepts were born and built into this version of ANNA: the Algorithm and the Checkpoint. To each built-in algorithm and checkpoint the core reserves a node. The developer may add more algorithms and checkpoints. To trigger a checkpoint, the developer places a system call in the code that's being traced to invoke the Physiology Analyzer. Once entered, the Physiology engine looks for the calling thread and the current resource in the projection and adds them if they're not there already. The monitor completes the arcs of the projection physiological graph if necessary and sets the new coordinates of the thread in terms of its state. We're now watching "live" the progress of complex algorithms in terms of its high-level state transitions on a graph.

The Physiology view has a mode where it shows a system's current multi-state as a multi-hand watch spinning around their threads. It has another mode where it discards of all thread identities and shows the Prediction Vector of a cluster of threads in the middle of a ring o resources. The view looks like a Praying Mantis with its legs onto the circle of resources.

Changes to the core to enable physiology monitoring:

  1. STREE_BASE::Lock and Unlock are new wrappers for the former DoInsertRemoveLock which we needed generalized and public.
  2. We've implemented 2 new proxy interception methods: Interlocked and CheckPoint to implement the Physiology Analyzer and its GUI frame. These 2 new channels use a different scheme of proxy updating where only the editors are notified, the list of proxies is not locked and therefore not presumed to be immutable.
  3. Calling BASE_CHECKPOINT(algorithm, checkpoint,flags) from certain HOST_OS methods under the HostOs predefined algorithm. The HostOs algorithm currently covers FileOpen, FileAccess, CloseHandle and Sleep. The algorithm is Markovian.
  4. The GOAL node was removed from THREAD. We kept CALL as the placeholder/scope of the DNA relations that a thread implements.
  5. THREAD::flInPhysiologyMonitor is a new flag used to prevent re-entry in the Physiology module due to interlocking.
  6. NODE::flHidden is a new flag that the thread proxies and the praying mantis of the Physiology view toggle to change from thread-oriented to prediction mode. Hidden nodes also hide their connecting lines. NODE::AutoWeight is virtual so that the GUI may reach it. LINE::vaHideNames is a new flag used in the Physiology graph.
  7. A new trick to draw two otherwise overlapping lines uniting two nodes in both directions apart from eachother.
  8. OBJECT::unprotected_locks and unprotected_unlocks are two new equilibrium indicators adjusted before the Lock/Unlock spinlocks.
  9. Enhanced the SetTypeMaster (destructor) code to better deal with mutual exclusion in proxy updates.
  10. OBJECT::Lock/Unlock now call the new Interlocked method at the MicroKernel level to perform physiological analysis.
  11. A series of tree-oriented functions were protected with Lock//Unlock as required by the Physiology system.
  12. A fix in GetSizeOfConstant so that the Physiology DNA thread passes the integrity tests.
  13. Core calls K5CHECKPOINT(algorithm,checkpoint,flags) to plot to the Physiology monitor.
  14. Added the new TIsNodeClient::flSelected flag to mark proxies for selection in a tree view.
  15. RenderBitmap and RenderEmf are the new public virtual methods of CAT_STREAM for use in GUI.
  16. TCanvasStream::Render is a new common handling code for both BMP and EMF. Significant rework of the canvas stream to allow schematics in contexts other than the Schematic Editor.

The Physiology viewer has the following components so far:

  1. The main schematic area to show the projection graph.
  2. List of threads / clients
  3. List of resources / states
  4. List of available algorithms
  5. Grid view of the graph with animation of current state and selection features
  6. Plot of the rate and bar-graph for the selection
  7. Status-bar for equilibrium values.
  8. Tool-bar with buttons for run, single-step, mode, etc.

The Physiology Analyzer is now the tool to single step through the OS on precise boundaries as defined by checkpoints.

From a constructive point of view:

  1. Physiology is an Enzyme thread recognizing the NODE, LINE site of the ABSTRACT nucleus
  2. Physiology has client, resource, metastate, link, path, direction nucleotides to seal itself from the genetic domain of the CORE.
  3. Physiology has its own canvas and DxD component to avoid re-entry through the system's memory manager and serializor.
  4. Physiology is a grammar-based dictionary of states and a limiting historical circular buffer.
  5. Physiology is a prediction engine for the next Markovian state of a system.
  6. Physiology implements an endomorphism from the cell upon itself.


311 Thursday, June 24, 2010

(done in v313) The plan is to achieve re-locatable growing sub-allocators. The main applications are dictionaries and the SQL database.

The relocation algorithm for sub-allocators with deactivating, moving, reactivating the heap doesn't work due to the links established from DB to Core which makes the core not traversable, in turn making the DB non-memory-scan-able. Brought back in an older algorithm that relocates the sub-heap by first giving it the free block of the master heap, then recursing the defragmentation logic, finally taking back the free block from the tail. This is OSMOSIS. This works as for defragmentation, but would not work for general relocation of heaps such as would definitely be required with heap growth. Both algorithms are in place and selectable through the "algorithm" local variable of OBJECT::Reallocate.

Enhanced some reflection methods to pass the SMapFormat pointer through.

LINE::SerializeLine had a problem with wires to a USE relation.

Fixed a crash condition in METACORE_API write_row to do with the thread current allocator not having been set. Some enhancements of the METACORE_API in the non-object row model. A database in this model is built as a super-type aggregating by value every table with cardinalities corresponding to the number of rows in each table. Much more work is needed here, and eventually this is how we want our DBs to look like.

There's a problem with where to put the ToDo's so they make sense...

TODO
{

  1. (done in V312) DB portal has a small amount of internal memory. Flood it to trigger a resize of the heap.
  2. MemGrow needs a direction flag. At the moment, MemGrow is limited to moving objects from right to left only. We must extend the algorithm to allow movement from bottom to the top.
  3. Allocator has a nucleus - the type instance handling the heap. The heap is currently deemed to be the space immediately following the nucleus, all the way to MaxSize. This design only yields allocators whose nuclei are located at the extreme left. Must enhance the design to allow the nucleus of an allocator to be anywhere within the cytoplasm, and even outside. We might need two pointer fields for the heap's limits instead of the ordinal MaxSize.
  4. Need a clearer API to move an object or an amorphous block to a destination address.
  5. Our first attempt to freeze a cell at its source position, copy it, and then defreeze it at destination corrupts the environment at the point of origin in a way that currently cannot be fixed by parsing the cell at destination. The issue here is with crossing morphology back-links. One possibility is to press harder to change the DNA fabric of K5 from TYPE+LINE relations to just connectors. All types would have distinct Scope, Input and Output bonds. When a cell is relocated, the environment is broken only as far as its inputs from the cell are concerned. All outputs are still traversable in the environment so parsing the cell becomes possible.
  6. Another source of concern is CreateInstance which is our "new" allocator with default constructor. The problem with it is that it doesn't allow the creation of any floating object that isn't at least a NODE.
  7. A database is a type multi-relation to its tables. Must rework the METACORE_API to seek, read, write, update, delete, add rows to any table while morphing this database type and its associated constant. Database is probably an auto-type.
  8. The whole core must persist no matter what objects it holds within. It must persist with one or more database cells, with one or more dictionaries, with auto-TIV.
  9. Need object bags and vectors. A vector is a series of adjacent structures of a given type all kept in an amorphous block.
  10. Specializing the two IO points away from the scope promises much faster reflection and canvas drawing. The directionality may resolve a multitude of existing issues. Nodes may gain IO scopes right now without changing the notion of a relation.
  11. Eventually, all core classes must be sub-allocators containing their relations and methods. The notion of a relation field different from its connector would go away in favor of memory containment. The DNA of the core would then become hierarchical. CLSID would go away. The VMT/TIV difference would vanish.
  12. Proxies must be obsoleted in favor of TypeInfo. A proxy is an instance of an object, objects become type masters for a more volatile persistence space. Proxy is distinguishable amongst instances by the volatility of its allocator. It implies that every proxy must be allocated in a managed heap, or must be flagged as such.

The steps to rework the core:

  1. We need to generate source-code from K5. All core source files must be controlled by the core. K5 must have a physical model of its sources. All H/CPP nodes of the core must be represented with stream nodes in the system's backbone. Must merge BASE and ABSTRACT directories. Must split all source files so that precisely one class per source file is allowed.
  2. Create the new type CONNECTOR descendent of TREE.
  3. Demote Pair from LINE to CONNECTOR.
  4. LINE descends from CONNECTOR.
  5. NODE descends from CONNECTOR.
  6. CONNECTOR has an endpoint count field.
  7. CONNECTOR is a vector of endpoints. The size of one like such is calculated from the count field.
  8. Change line connection logic. Change the traversal logic across the core. Make sure everything still works.
  9. (done in v313) Get back to relocating databases using the freeze-copy-unfreeze logic.
  10. Rework the core based on a DDNA of simple connectors rather than relations.

}


312 Thursday, July 01, 2010

I'm experimenting with a metamorphic cell as the database persistence instead of the simpler PORTAL. The reason is type-independence. A database may be relocatable if it has its own full DNA copy to be pointer independent from the surrounding operating system. The following DB operations work fine now with the FMDI (freeze-move-defreeze-inform relocation) scheme:

drop database test; create database test; use test;create table anna_k5_gui (dummy int);
drop table second; create table second (i int, j int, name text);insert into second (i,j,name) values (3,30,'mihail');
Force a defragmentation of the core...
insert into second (i,j,name) values (4,40,'mihail4');
select * from second;
... with the expected 2 results.
... Force another defragmentation of the core...
select * from second;
... with the same expected 2 results.

The same works with relocation algorithm 1 (recursive defragmentation) as well, but the core fails the integrity test with a crash on an object's name dereference. With DB as PORTAL, algorithm FMDI fails due to the reverse environment cross-reference, algorithm RD works, but the same integrity fault occurs as with DB as ABSTRACT. Both paradigms of the DB, as PROTAL or as ABSTRACT pass the self test after relocation with RD algorithm only if the Algorithms and Checkpoints are not present in the core's map.

Resolved the failures of relocations with Algorithms, Checkpoints and CODESEGMENT enabled. The problem was that in ABSTRACT::Relocate the components must be called to relocate in the exact order of declaration.

Relocation fails now only when formats are enabled for Methods and Parameters. The UML loader also fails when CODESEGMENT is defined.

There was a mapping problem: the core only has 679 methods as defined in formats.cpp. The correction has allowed the core to relocate the DISK using the Recursive Defragment algorithm. This is an important result.

AutoTIV building is done once by calling AbstractKernel->Analyze(0,0);

UML can be loaded on a core with methods and parameters defined, with AutoTIV on, but whithout CODESEGMENT as there is a problem with the OPCODE autotype. The core must also be set on the RD relocation algorithm for UML to load.

Enhanced the ABSTRACT nucleus with RPC format information:

  1. Needed some fixes to output of Gaius Correlator so that the MRF output is somewhat more usable and more correct. The script was placed into the new ABSTRACT/formats.cpp file which is included by the core's constructor. The format.cpp file holds all the methods of the core and their parameters.
  2. Enhanced the nucleus with Methods and Parameters. The current V312 counts are 681 core methods and 2034 parameters.
  3. CODESG and OPCODE are two new type masters in the nucleus. ".code" is an unbound constant of the CODESEG type in the allocation of the nucleus. The UML loader can use those during binding.
  4. Layout had to change.

Preparation for heap growth:

  1. MemRealloc must prepare for heap relocation which means shifting the this pointer. It's logic moved to a new static method TYPE::MemOperation so that the self pointer may be updated. MemGrow is really an extension of MemRealloc so it too is now static like MemOperation.
  2. Multiple memory allocators within the morphic core.
  3. Made changes to the size calculations involving USES relations so that DISK and Database are USES auto-relations of their scope.
  4. The K5NULL constant is now specific to a given cell nucleus.
  5. CAT_MORPH::CreateCell is new and used to both create the DISK cell and also by METACORE_API to create the database cell.
  6. The ABSTRACT nucleus now constructs namelessly with CAT_ABSTRACT::InitializeStructure from both INIT and SetTypeMaster. INIT applies names to the structure through the new CAT_ABSTRACT::SetNames call. We can now create isomorphic nameless cells with the default constructor.
  7. A blank cell is an ABSTRACT nucleus with a heap containing the amorphous blocks for the constants.

Another deadly battle to achieve relocation of memory managers.

  1. I'm now working with entire metamorphic cells. Those were extended to allow for an internal heap.
  2. The handle of an abstract nucleus is set to right after the nucleus into the start of the cytoplasm. To relocating such a handle the RELOCATION_SHIFT_INTERNAL had to change to consider that the [oldo,X] interval is closed to the right as well.
  3. Found a source of trouble in OBJECT::IsType where a an autotype was rejected as not being of its own type, which was a contradiction.
  4. NODE::GetInstanceSize and TYPE::GetSizeOfConstant call everything else statically.
  5. TYPE::MemRellocate takses the start and last_address arguments and also promotes the protection information so that it may work on protected ranges as well. TYPE::DoRelocation also takes the protection information argument. We had to have this for the freeze-move-defreeze-inform relocation scheme (FMDI).
  6. Extended the protection mechanism with another function protected_update to update a pointer when a memory range is protected. TYPE::DoRelocation updates the bonds of an object using protected_update.
  7. Found and fixed a fault in getting the object's type in MemRellocate command==1.
  8. A new TYPE::MemResetCache that doesn't rely on tree chaining, but rather it scans the heap from start to the end looking for the first free block. This is to support sub_allocator relocation algorithm=0.
  9. Found and fixed problems in SerializeHeap with stepping over ABSTRACT nuclei and over user types.TYPE::SerializeHeap doesn't clear the proxies anymore. Doing so was stalling the table proxies from the METACORE_API during relocation.

Found and fixed a DNA mapping error in CAT_SQL with the SELECT statement deemed to be a TYPE descendent on one hand, but generating NODE on the other. The problem lead to the errant binding of Edit_TYPE to all NODE descendents which meant that editors couldn't open on simple constants.

Found that HOST_OS_RT::SetTimeReference did not work to calculate deltaT and always returned the full period since the reference was taken. The correction now yields some strange results on the Physiology viewer where only the diagonal ever has nonzero time values, and lines never do.

Added the effort indicator in the status bar of Type Viewer. Need this to check if there's any improvement in the morphology scan.

Re-instated the DxD retained mode demo.

Profiles for a format scan of the ABSTRACT nucleus with full method and parameters info:

  1. GetMem(0) effort : 0
  2. List mode GetMem(mf) effort= 78,480s
  3. Core formats: 619382
  4. Size of the core: 1649095 B.
  5. Defragmentation: failing with both algorithms.
  6. Redraw effort with names and lines: 0.8s per frame.
  7. Redraw effort without names: 0.65s per frame.
  8. Redraw effort without lines: 0.357s per frame.
  9. Redraw effort without name or lines: 0.146s per frame.

Started a new line of work to speed-up morphology traversals and to make relocation safer:

  1. Two new virtual methods for OBJECT, NODE and LINE: IsNode and IsLine. These are meant to provide an immediate answer back to the traversal process without scanning the morphology which may be inconsistent at the time.
  2. When NEW_IO is defined, the Pair bonds of LINE get demoted to TREE. The Pair will then become a node's IO insertion points for connectors, so there's no need for those convoluted filter loops.
  3. Relation fields are objects with the flField set, which is safer to test than to scan the morphology.
  4. Amazingly the system mostly works under NEW_IO, but relocation fails and the memory maps have gaps everywhere an auto-relation is inserted. The reason for these holes is that relation fields in this model cannot induce size into its parent. To change this, an entire redesign of the core's DNA is required.


313 Sunday, July 04, 2010 (Hierarchical Memory Manager with frontier negotiation)

Relocation of complex structures has been resolved around this time last year in version 271. Relocation of sub-allocators is the generalization of that principle. Let (A) an allocator be a space partition of a base allocator (B). The issue is inflating, moving, deflating allocator A within its base B while preserving the integrity of its internally allocated objects and those of B. Relocating allocators and frontier negotiation has been a major engineering goal since ANNA K4 and a milestone foreseen in our self-imposed SRS at Features. It's only now after 10 years of the most intense work and a 6-fold redesign that the objective was finally reached.

The answer to the matter of relocating allocators within a hierarchical storage space lies in the DNA of the system. To relocate one must have a certain minimal DNA to enable oneself to be type independent from the rest of the system. Having autonomous DNA, the allocator becomes a cell which is polymorphous with the system but topologically independent from it. The problem with relocation is that freezing a structure at the place of origin makes the entire environment non-traversable, which is a critical state. Giving the object its own DNA makes it less likely to need links to the environment.

We demonstrate/apply the relocation principles with our SQL driver. The metacore now creates a user session as a cell with an abstract nucleus and with a minimally sized cytoplasm. A database is a typeset that gets assembled in this user-space. The table types then instantiate records. The instances move around within the user portal and the tables are metamorphic in the way columns are added and removed. The portal is therefore a nucleated metamorphic cell polymorphous with the system's kernel. Another long-standing objective was therefore reached, that of the Object-Oriented Database Engine.

With the relocation and growth of allocators within the organic soup this core makes an important step forward. We were able to inflate the database cell on a per request basis. On Unbelievably, on simple inserts most of the memory management functions were transparent to the relocation of the memory manager. The default constructor was the only piece of code that had to adjust after allocating memory, and it now does so by using a proxy to its memory manager. The matters got more complicated when a complex database with several tables was assembled. Several of the core constructors had to be revised so that their context was adjusted through the use of proxies to shifting objects. The relocator updates any back-links that the interlock or the proxy genes may establish. Taking a proxy is the best way to reference an object which is sliding through space. We are well aware that the entire call system has to change and that the matter of relocation would only be fully resolved when full method morphology and auto-trace would be in place. We also need to rethink the notion of a USES relation to be a general link to a category plus an offset into that category over to the object used from that category. Metamorphosis would have to update those offsets. Clients of USES relations would not insert traversable relation endpoints into the objects themselves, thus avoiding the freezing state discussed above. We would also need an overhaul of selectors.

Implementation details for Inflating Memory Allocators:

  1. OBJECT::Grow changes the content of an instance. It was enhanced to sense a sub-allocator and relocate it to a larger space by doing FMDI (freeze-move-defreeze-inform). This algorithm is of intense subtlety. It first freezes the instance with SerializeHeap (which is really a misnomer and should be changed to HeapScan) using a protection range that excludes the bottom amorphous bond. It then demotes the heap to an amorphous used block. It reallocates the used amorphous block to a new and larger place (or in-place). It promotes the amorphous block back to heap level. It then de-freezes the heap and finally informs the environment of the relocation having taken place through echo-location. Once Neither the underlying heap is not safe yet unless the growing object updates its MaxSize to the new extent. A left-to-tight defragmentation is carried-out over the current heap to condense its content to the bottom and reveal all the free space at the top in one single block. Only then the size of that free block can be adjusted with the given growth delta. This defragmentation step could be avoided if the heap ends with a free block already or if the growth factor is larger than an amorphous block. It is also probable that one has gotten here due to a short-falling memory request which had already defragmented the heap.
  2. TYPE::Grow was renamed to TYPE::Inflate as it belongs to the metamorphism algorithm rather than to the relocation algorithm, thus exposing the underlying OBJECT::Grow method used for free space injections.
  3. Any method that does inflating memory allocation must expect object pointer sliding and must therefore reintegrate its context using any given method ranging from taking a series of proxies to those objects before inflating, down to finding those objects by traversal.
  4. All constructors should expect relocations to occur within the memory manager in which they create the instance. OBJECT_BASE::property_Name is one such method which triggers relocations. Another method taking proxies is TYPE::CreateInstance and uses a proxy in its local context to track the shifting of the "self" pointer during relocation.
  5. The constructors of the abstract nucleus take proxies to this and use the explicit self pointer for de-references.
  6. TYPE::Truncate is a short for a negative OBJECT::Grow that leaves one byte of free space on the heap, and demotes the remaining space down to the underlying allocator. The new method is triggered manually from the MapMem GUI. Truncate is a stream-oriented method, doing on heaps the same as it does on files.
  7. Enhanced the GUI to ignore any timer-based refresh of controls on inactive tab pages. The Schematic Editor is quieter, so is the closure monitor. All GUI editors now route through the GetServer() of the proxy which is safer with relocations.
  8. The __LMG loop of MemAlloc is there to make the memory manager loop around MemGow to return a free block which is large enough. The loop exited and the allocation request was deemed a failure is the allocator was found to be full (no free blocks). The new logic is to attempt a downward inflation of the nested allocators.
  9. Found and fixed a looping problem in TYPE::MemGrow.
  10. TYPE::MemGrow on current allocator (A) calls OBJECT::Grow with the given delta which in turn may go down to any other base allocators with MemGrow recursively. It locks the base allocator in descend and unlocks it on backtracking.
  11. TYPE::MemLocateBlocks(L,obj,R) is a new method to concisely get the two amorphous blocks that flank a given object. For any given object, a pass to the right is done with TypeScan over any other adjacent objects until the first amorphous block is located. The back-link of that block is the amorphous block which is closest to the given object on the left. These two blocks are needed in TYPE::Reclaim and the new TYPE::Truncate methods which insert amorphous blocks back in the heap's chain. Remember that ANNA's heap is a chain of amorphous blocks interwoven with trains of objects that are not in the heap's instance list.
  12. MemResetCache changes toe hovering children pointer of the memory manager to be the first free block at the bottom of the heap. It returns this block if one is found, or the first used block if the heap is full. If there's no free block, reset leaves the hovering pointer untouched.
  13. The implicit this pointer is unfortunately not reloadable (C++ language constraints) and a self alias needs to be used instead. Several memory management methods have become static for this reason. TYPE::GetSize was a misnomer, the method having a much more general role in the system as the morphology iterator. Turned GetSize into the new static TYPE::ScanType which is also relocation-sensitive around the callback, and keeping GetSize as a virtual shorthand for any external uses. Remember that the meta-core may only be used via its virtual methods.
  14. The role of TYPE::UpdateInstance is clearer now. This method implements ANNA's translation processes. Transcription of a foreign mRNA may cause cell inflation and relocation so the method is now using a proxy to track the instance in its local context.
  15. A problem was found and fixed with GetServer() which should not rely on a valid Object pointer, rather return the top of the object containing the Proxies.
  16. TForm_EditFrame was wrong in taking copies of the Object that is being edited. The direct pointer could not be updated by the core if relocations occurred. GUI editor sheets now route through a new property to the form's unique proxy object.

Enhanced TForm_TYPE_MemMap with new controls to inflate a memory manager by a specified amount of free space and to trim any free space. Trimming is done by splitting the internal free block into one minimal, and another maximal which is then given to the super allocator. Defragmentation must never starve the allocator. Trim and Inject are threaded actions so they may interlock onto the memory managers.

TODO

  1. Persist to BSS a core with multiple cells with complex contents.
  2. Assign Methods and Parameters to CAT_MORPH to not impact so much on the size of the nuclei.
  3. Verify and correct the destruction of cells.
  4. Re-balance the closure monitors during core initialization.


314 Wednesday, July 07, 2010

A new scheme of LINE to NODE binding is in place under the NEW_IO switch. It dramatically speeds-up type scanning. TREE was extended with the new Ins bond while LINE only extends on that by adding the Outs bond. Lines insert from a node's scope to another node's Ins. This asymmetrical architecture limits the backward traversal which we set about preventing for relocations using the Freeze-Move-Defreeze-Inform method. This was the answer to having auto-relation fields still induce size into their parent class. It's hard to appreciate it yet, but this could well be an important reform of the core.

  1. Moved Methods and Parameters from ABSTRACT to MORPH so that they do not impact on the size of the nuclei.
  2. Changes to Gaius to drive the generation process into the core's serializor. Larger stack size for the code generator.
  3. Added the EditAdjustIndent function to HOST_OS.CPP to support the generation of source code using the core's serializor.
  4. Some protections in STREAM::SerializeExport: The Micro Kernel, any thread, the current stream should be regarded as exports.
  5. Found a discrepancy between the OBJECT_BASE::EVmt and the actual VMT causing the serializor to crash.
  6. Want to hide the Name field of objects so that it may be encoded within the name pointer itself, thus reducing memory consumption. Work is in progress. Added the get_Name method to OBJECT_BASE in preparation.
  7. LINE::ofHideEndpoints is new and used to profile the time spent drawing. Drawing endpoints is 25% of drawing.
  8. Trying to avoid rotation when the angle is 0.
  9. Natural left-to-right paths in Type Viewer.
  10. Analyze is currently time-consuming as it serializes method signatures for textual comparison. MORPH now calls Analyze only on the Abstract branch to minimize the effort.
  11. Tuned METACORE_API to commute from a PORTAL to an ABSTRACT session at the flick of a switch. I'm back to experimenting with the relocation of portals when they've got links to their environment.


315 Wednesday, July 14, 2010

Database is back to being a PORTAL which is simpler than a full abstract cell. Portal doesn't have its own DNA. Relocation and growth now work over such allocators as well.

  1. The new TYPE::Membrane method is a translation that iterates an allocator's interface with its environment. Membrane can be instructed to perform a relocation update.
  2. The new sub-heap relocator calls the new membrane function to echo-locate the environment after de-freezing the heap avoiding those discontinuities of updating the head first, then parsing the heap, etc. The new logic replaces the former body of TYPE::MemRelocate which was a mess.
  3. There are relocation problems with the outbound strings of a sub-allocator. PORTAL had to allocate its name within its own space in order to be relocatable with the latest algorithm.
  4. SMapFormat has a new flags field with two flags defined so far: flIsObject which SerializeHeap sets to distinguish between amorphous blocks and objects, and flNoProxies that Membrane sets so that the returned frontier doesn't include the vanquished proxies set during ScanType.


316 Friday, July 16, 2010

Tiger2 portal is now embedded into the resources of the nucleus.

  1. Metacore starts the portal right after it creates the user session so that ANNA may be traversed by HTTP clients even when running as a module of a service.
  2. HOST_OS::SizeOfResource is a new api.
  3. Found and fixed an error with processing the carriage character in HOST_OS_COMMS::TransferLine which prevented the HTTP protocol from correctly draining the socket of all the pragmas that IE sends us.
  4. Threads spawned by the PORTAL listener now use the portal as memory manager. The entire activity of a portal is therefore confined to the allocation space of the given server.
  5. The login portal, which is the counterpart of the server portal, is now a memory manager and must therefore have the minimal 1 byte free block from which to grow.
  6. Tiger2 isn't issuing the "No information available" string anymore.
  7. The last resort in locating a web page or an image is to search the resource pool of the system. Tiger2 is now embedded into the resources.
  8. Fixed the depth problem in building the cross references for the center information panel of the WEB portal.
  9. Added a new algorithm node for the allocator together with its checkpoints. Increased the size of the nucleus to fit the new constants and strings.
  10. Added a nesting prevention statement in Physiology::Checkpoint.
  11. Fixed a long standing problem with the closure monitors all going wrong when a stream is closed while it still owns locks. ReleaseSublocks is a new method to be called in advance of freeing a stream with sub-locks.
  12. Avoiding an unnecessary unlock of the canvas in RenderEmf.
  13. Decided that the relocation algorithm should not lock onto the objects that it relocates. The deadlocks that would occur would be unmanageable.
  14. INIT_THREAD cannot call GetSize as its first call in advance of setting-up the thread.
  15. Every amorphous block is now born bound to its allocator so it is always owned. Low level malloc and realloc could not otherwise deal with block string ownership.

Fixes to Tiger2

  1. Tree.htc onclick now avoids reloading the same node into the correlator.
  2. Panel.htc failed when turning from text to diagram mode when no item was in focus.
  3. Panel is somewhat of a tree view (which in turn is more of a schematic) so it also needs the SelectedItem for tracking. Corrections to the OnClick and OnDoubleClick event handlers to allow the center panel to drive the navigation through its hyperlinks.

Fixes to shutdown procedures:

  1. Autobind can and should be on most of the way throughout the destruction of the core exactly symmetrical as during core construction.
  2. Deleting the MORPH category must be done with a DO_DONE like sequence so that the cables of MORPH are removed from their endpoints before deleting their contained wires..
  3. TREE::DONE needed more convoluted logic to prevent recursive destruction of scope children when the node or the line is inserted into the null constant.
  4. Starting and stopping the database portal is now done through the creation/dropping of the abstract anna_k5_www table.
  5. PORTAL::Start takes the "rename" parameter which, if set, renames the portal to the name of the machine. Once a portal is renamed, it couldn't be located in the metacore api table create. We're now preventing the rename so that the WWW server may be started by creating a dummy table called anna_k5_www. Deleting this table sends the quit command to the HTTP server.
  6. Better logic in PORTAL::Stop to send the QUIT command to the listener via a command socket. Logic to sense whether the port is "silent" or not.
  7. There's a new function in HostOs.CPP called CloseAllSockets which forcefully closes all socket connections.
  8. Configuring any secondary nuclei at the top of InitializeStructure so that the relation (self,self->Relation) can be bound without triggering integrity errors.
  9. Rework of the core to not flood the log with closure errors or with line integrity errors.
  10. Core INIT is now done on a temporary wrapper thread who lives between the end of the initialization of the nucleus and the end of CAT_MORPH::INIT. This is a place where more complex initialization may take place. The init may now enter interlocking arrangements. The shutdown is unfortunately not quite symmetrical.
  11. Have to delete all instances of ABSTRACT before deleting the ABSTRACT type master itself. Ideally, the apoptosis of the OS should delete everything other than the nucleus before destroying the nucleus.

This version is exiting cleanly in all test scenarios.


317 Tuesday, July 20, 2010

A few changes and corrections to restore the BSS persistence functionality. The MORPH category wasn't mapped correctly and had overlapping regions. Such a structure could not be saved and loaded. The sign of SMapFormat.Delta had changed during the last modifications to the pointer virtualization mechanism. The MORPH-ABSTRACT cable has to be controlled and mapped internally so that Morph.T.Children never falls outside the self controlled region of the metamorphic cell. Added a new method to list the cables of the cell. Some fixes to TYPE::SerializeHeap to allow storing and loading from BSS files.

This version stores to and loads from BSS but corrupts the content of any sub-allocator such as dictionaries, or portals if any are present.

Fixes to the Schematic Editor to lock and unlock the canvas for operations that would require memory allocation. Bending a cable in the nucleus was a deadlock in the previous version because the abstract nucleus would be locked to the canvas, a bending thread would lock the nucleus for memory allocation, but the canvas would not be locked to any thread and was therefore treated as an owner blocking the second thread. Canvas needs to be locked prior to any use.

Changes to the Schematic Editor to use IsLine and IsNode virtuals instead of the complex calls to parse the morphology. Changes to the type factorization routines of multiple selection to always resolve to either the NODE or the LINE type. There's still a problem selecting lines, though.

Some space has to be excluded from the stack at it's bottom. Experimenting with the value, currently at 0x4000 thread creation fails. 0x6000 is okay.


318 Tuesday, August 01, 2010

Enhancements and corrections for Tiger 3 Morphology Server.

  1. The greatest problem of Tiger3 was the size of the diagrams to the tune of several megabytes which rendered the morphology portal inoperable over the internet, thus defeating its purpose. Diagrams had to be compressed.
  2. Found that the prototypes of fclose and fdopen of our ANSI library were wrong and fixed them.
  3. Integrated ZLIB into the base micro kernel, much like an extension to the ANSI library.
  4. Tuned the ZLIB deflate algorithm to produce GZIP wrapped compressed output with a ratio of some 20:1 which in terms of download speed is 20 times faster.
  5. Had to change the HTC DHTML components to use another proprietary technology from Microsoft in Internet Explorer, namely VML, which is available as of IE6, or so. Every page of the Tiger3 portal has to enable the v namespace with the v\: and the .shape styles.
  6. The HTTP core server now embeds a diagram as a v:shape tag with v:imagedata as diagram.emz which is compressed enhanced metafile.
  7. When Tiger requests the construction of a diagram.emz, PORTAL first serializes an EMF, then calls the ZLIB gzip compressor through the new METACORE_API::Compressor method. Since neither the compression ratio nor the size of the output buffer can be estimated before compressing the data, PORTAL must allocate a destination buffer of equal size to that of the source which is expensive. It would have been much nicer if Windows GDI provided an API for rendering onto a compressed EMF device right-out of the box. Consequently, the portal for a tiger server must provide some 2*16MB of heap space for these buffers which is horribly expensive for our miniature metacore.We might have to deal with this compression process entirely outside the managed scope of the metacore to charge the host OS with the memory expense.
  8. Had another piece of trouble when merging the response of the Url.HTM response into the div of a panel in that the diagram was fetched twice: once when the Url.HTM was received, and once again when the span was merged into the DIV. To avoid this behaviour, the core portal now sends the Url.htm with the diagram commented out and the merge logic in the HTC client removes the comment. The result is now that the diagram is fetched only after the HTML merge providing for the fastest possible diagram download.
  9. Learning how to integrate this hardly documented VML into our Tiger server was painful.
  10. Enhanced Tiger HTTP server to do horizontal resizing.
  11. Disabled all aspects other than the morphology of Tiger.
  12. Default mode of infra and supra side panels is now diagram with a depth level of 1.
  13. Auto-centering the diagram after zooming in or out.
  14. Enhanced CanvasStream to initialize a font. Tiling is now possible in the absence of the Schematic Editor. Tiger3 can now render correct diagrams without a GUI front-end loaded.
  15. Core auto-tiles the schematics after construction in the absence of a GUI.
  16. A fix to SE's ConeSelection. Behavior is still wrong.
  17. STREAM::Nest blocking in a socket Accept was prone to morphism and had to use a proxy object to track the stream of the parent stream.
  18. Cables cannot be initialized withowt a map that Gaius should produce. Had to be disabled.
  19. Deeper stacks in PORTAL's communication threads to allow temporary storage of buffers.
  20. PORTAL responds to diagram pick correlation requests.
  21. Found and fixed a problem with the coordinate system of the Render method in Tiger/Portal where the bounding rectangle had to be translated over to the center of the viewport (canvas bounds).
  22. The bitmap of a 16x16 node should be drawn from 0,0 instead of 1,1. More should be done to draw the (empty) selection rectangle over the top of a node's face.
  23. Changed HOME to USERS which is the proper name in keeping with what the namespace stands for.
  24. The second (passive) nucleus was renamed from DISK to CELL and is currently linked in from users logging into the built-in Tiger3 web server as a demonstration structure.
  25. Added some URLs to the descriptions of top category.
  26. A very subtle problem with Internet Explorer. A page that had <a> links to "HTTP user IP:PORT" would be deleted from the cache and not loaded. Changed the name of our protocol to WWW instead of HTTP to avoid the issue. 12H of tracing. The problem was preventing the logging of certain trees in the IO panels of Tiger.
  27. Server console messages no longer contain the user's information field which may now be a huge HTML script.
  28. Added PORTAL::CreateDemoStructure to configure the user upon login. Each user gets its information field set to our welcome and copyright notice. The user gets a sub-node with the help page drawn from resources. The user is linked to the passive CELL structure for demonstration. A demo structure of 6 nodes is built to stand as the LEGEND and to allow the user to pick and navigate and get acquainted with the workings of our K5 system.

Enhancements to TIGER:

  1. Pick correlation on diagrams is the most important enhancement in this version.
  2. Picking is done with PANEL::LoadPanel("P")
  3. Added the error message for browsers other than IExplorer.
  4. Added hints to every button and input line.
  5. Info panel can track internal URLs and external www sites if anchor tags are present in the information fields of the navigation structure. External is any URL that does not start with the current location of the main page.
  6. Foreign sites show-up in the info panel where requested.
  7. Pecking at the external site is not possible due to IE domain security.
  8. Got the Navigator to reload the center panel into schematic mode. A double load is required.
  9. Removed some Panel OnScroll logic that caused a fault in the browser.
  10. Added mailto support to the bottom bar.
  11. Added the Help.htm page to the Tiger3 embedded web server.

Problems with this version:

  1. Core destruction crashes on CABLES as they are not initialized.
  2. Cone selection in SE is erratic.
  3. Integrity errors after a persistence cycle in node points vector ownership. Node doesn't own its points cache DISK,(null)
  4. Tiger fetch occasionally hangs in an endless file write on the socket of the portal.
  5. No recursive relocation on portal-user session.
  6. HTTP portal of 32MB for diagrams is prohibitive.
  7. Crashes after persistence cycle relocation when opening editors.
  8. Defragmenting the core while a HTTP server is started is okay, but server crashes on calloc during the first accept. It seems that the listener thread is not being updated along its MemoryAllocator field.
  9. Need THREAD::SelfTest to see that MemoryAllocator field is valid.


319 Saturday, August 07, 2010

Changes to make the system run on FACTORY which is a single CPU machine:

  1. Found a crash condition in the destructor of the Schematic Editor where the canvas cleared its font. The initial canvas font is owned by the GUI memory manager and must not be passed into the core for destruction or relocation.
  2. Found what was hogging the CPU all this time: our GUI loops have to block on WaitMessage().
  3. PORTAL sets its user-name only if it was created with some internal heap.

Added some handling of the POST command in the PORTAL http handler.
Enhanced Tiger with the INFO.Transfer routine to open a HTTP connection from the client upon the server and send the content of an INPUT element. We aim at transferring an XSD file over to the Morphology Server for processing. Unfortunately, the socket only works in URL mode and sends translated data which the sever needs to restore. XMLHttpRequest.sendAsBinary doesn't work.
Added the LoadXML and the AWB external commands to the monolithic build of ANNA.
Enhanced the Schematic Editor with another action to create a type with internal free space.
Corrections to the refresh timer of the Schematic Editor so that the destruction of the refresh thread isn't triggering another refresh. The SE and the scope viewer are now quiet.


320 Saturday, August 07, 2010 (milestone : TIGER converts XML and XSD)

An important new version: Tiger3 now performs a useful task in that it now allows the web client to upload a schema definition files or an XML and parse it into the backbone, then lay out its diagram.

  1. Servers are now placed into the core's new Servers namespace. Initializing a default server instance into the system's Servers group.
  2. User is created with a 4M internal space.
  3. The XML_Compiler was moved from the external LoadXML.CMD tool and into the core as a feature of our HTML compiler. The external command drives processing through the inbuilt logic.
  4. Implemented the POST protocol. Had to pass the content length into the HandleURL method of PORTAL.
  5. The WWW user may now use the PIPE folder to post a file to the server. The data is grafted onto the scope tree as a MEMORY_STREAM node. The file is then compiled with HTML::XML_Compiler and the diagram for the user is re-arranged. A message is sent back to the AJAX user.
  6. Found and fixed an error in handling reads from MEMORY_STREAM: the data buffer must not inflate on reading beyond the end of the memory file.
  7. A fix to building the tiny library of the core.
  8. A quota of 256K for the XML file a user can upload. Checking the signatures for the presence of either xml or soap tags.
  9. Deleting the previous structure before compiling another XML.
  10. Some fixes to crashes in the XML_Compiler when simpleType elements are not present in the XML.

TIGER

  1. Implements the HTTP POST command in binary mode to send the content of a TEXTARE over through the ftPIPE onto the server where it becomes a compile-able memory stream.
  2. Succeeded to retrieve the server response over the XMLHttpRequest socket.
  3. XML compilation results in an arborescence grafted into the pipe. The Navigator panels must be reloaded so that the user is informed as to the effect of his submission. The LoadAll method of the navigator is now public.
  4. The client and the server text controls are auto-scrolling and they extend as wide as the parent DIV.
  5. A fix to the height calculation for the Morphology navigator which had the center panel shorter than the tree views on the side.
  6. Added dragging events onto the panel mode changing buttons where the height of the correlation strips may (awkwardly) be (finally) adjusted.
  7. Added some help to the ftPIPE Added a demo structure into the value of the text area.
  8. Navigator's center panel defaults to a depth level of 2 so we may see the whole legend first-off.


321 Monday, August 14, 2010

Fixes to the system:

  1. Corrected a long-standing problem in Type Viewer with clearing the lines of the grid. Attempting to clear the lines that are out of the visibility window would actually first trigger their creation. The map of the OS is in the range of 1/2 million formats and creating that many lines would crash the external memory manager. Avoiding the creation of empty lines by only clearing those in the visible range. Type View can now browse indefinitely large structures such as the OS or the CODESEG.
  2. Repaired the search capabilities of the Scope Viewer which were damaged when the layout of the proxies tree changed and the recursive descent of ScopeFindExpand could locate the root. The function now iterates the visible TreeNodes, at the last level of recursion, to find the proxy to the morphic core.
  3. Fixed a crash condition in the Schematic Editor that prevented the deletion of nodes.
  4. Removed the CAT_ABSTRACT::Type pivot throughout the system.
  5. The resource CLSIDs of the OTP core are now taken from the global uids.h file for consistency.
  6. Promoted strnchr from HTML to HOST_OS so others may use it.
  7. Added EditAdjustCharFormat and EditChangeFgColor as new RTF Edit APIs to abstract the EditorStream away from Windows. Increased the indentation factor (measured in twips) in EditAdjustIndent.


322 Monday, August 30, 2010 (source code generator)

This version of the metacore generates C++ source-code that is re-compile-able with external C++ tools.

  1. There's a new Compile(...) virtual method starting from the TREE type which implements all manner of text serialization. There's also the new STREAM::CompilePObj as the entry-point to the compiler/decompiler.
  2. The Compiler is really a Serialize method, but they should be kept separate during this phase of the development to not interfere with the already finicky logic of the existing binary/canvas serializor.
  3. STREAM has new methods to help abstract the generation of source-code over different types of streams. STREAM declares the new Bold, NewLine, TextLink, and SetColors methods.
  4. EStyle, ECompilePhase, ECompileFlags are new enumerators and flags to drive the new compiler.
  5. The compiler is multi-pass with possible passes listed in ECompilerPhase.
  6. Temporarily resurrected the old source-code compiler logic of the core under the OLD_LOADER_STORER define switch: STREAM::Get, OBJECT::Load/Store, TREE::LoadChildren/StoreChildren/LoadChild/StoreChild, NODE::(StoreChildren, LoadChild, StoreChild)
  7. Streams are created with Visibility set to Implementation so as to avoid listing them in the generated source-code. Threads are also of the Implementation visibility by default. CreateInstance generates Implementation objects which may need to be made public afterwards.
  8. Added two new stream types to the STREAMS category: RTF_STREAM which is still under development should generate an RTF stream in memory without animating a RichEdit control of Windows; HTML_STREAM was necessary to render the source-code with line breaks, bold, hyperlinks and colors for WEB presentations. These two stream types descend from MEMORY_STREAM.
  9. EditorStream::SetIndentLevel does not place opening/closing braces. WriteBuf had to be implemented to copy the data to a local buffer and emit it with WriteStr. EditorStream now implements the new Bold, NewLine, and SetColors abstract methods of STREAM.
  10. TIGER allows for negative TLS values to drive the information building serialization process through the core's source-code generator over a HTML stream.
  11. The order of methods in the UML model is not canonical. The .code constant constructed during the AutoMethods pass must either be relayed or re-created to match the ordering of the core. Simply setting during Merge the core's method origins and cardinality to the values of those loaded from the model would not produce a valid map of the CODESEG type. A solution is to extract during merge the text of each method, collapse/destroy the source method, then inflate the destination method with the locally saved text. Metamorphosis of the CODESEG, however, only seems to work in the growth sense and incorrectly for shrinking operations, a complex issue that needs further investigation. The work-around for the meanwhile is to let the core have its CODESEG type and have the UML loader use its own separate UMLCODESEG and .code constant. The approach is also more sensible in the general case of loading arbitrary user models which should have nothing to do with the core's text. The NODE::Merge procedure inflates the core's code, copies each method body, avoids the deflation of the UMLCODESEG before destroying the source method by forcing its cardinality to 0.
  12. The initial .code constant of CODESEG has 1byte of extra size to hold the string terminator and avoid the overlapping problem that the current metamorphosis algorithm seems to have.
  13. OPCODE has been erased from the core as methods are now Char relations.
  14. Re-ordered the types of ABSTRACT so that each class declaration precedes those that specialize it. This is to avoid forward declarations at the inheritance level which would otherwise lead to un-compile-able C++ code.
  15. ANNA's fundamental types are sets of constants that define the type's maximal size. The C++ source-code generator de-compiles these fundamental scalar types as enums with constants as as the enum fields. The external C++ compiler must then use a scheme of sizing the enums according to their maximum constant to generate executables. Anna currently uses the fixed-size 32bit integral enum. 64Bit enumerators cannot currently be compiled by Borland tools. Therefore, Anna's 64bit fundamental types cannot be de-compiled to enums and will not be de-compiled at all. Their internal names are identical to those used by an ANSI C+++ compiler. Int64, Double and DateTime are therefore deemed to be at an implementation visibility level. Any node set to implementation visibility would not be decompiled.
  16. Constants true/false were renamed to True/False to not conflict with C++. Bool is one of Anna's 8bit-wide scalar fundamental type that de-compiles to an enum.
  17. The C++ generator would avoid declaring any of the ANSI scalar types.
  18. Nodes such as Algorithms and Checkpoints in the ABSTRACT namespace are at Implementation visibility so as to not be generated in the C++ sources.
  19. Anna's categories such as ABSTRACT, STREAMS, OTPCORE, etc are now flagged as such so that they may be decompiled to a namespace for C++.
  20. USER_GROUP is gone. Servers and Users are both generators of portals.
  21. Drawing Board is renamed to TEMP.
  22. Fixes to the UML loader: static methods are not virtual, there was a VMT mapping problem here. Methods with ellipsis bind their last argument to Void.
  23. UML Merge drops float over Float, bool over Bool, long over Int, unsigned int over Int, ... over Void, and time_t over Time. AutoMergeOTPCORE is new.
  24. Reworked the CLSIDs to manage the gaps more easily.
  25. OBJECT::StoreComment returns the number of lines emitted for the comment. Client logic may want to avoid a NewLine separator for a single-line comment.
  26. An morphic auto-type is a relation.
  27. TYPE::IsStatic is a new virtual overloaded by OPERATOR and used instead of the direct flag test to say that static methods are actually inducing size into their CODESEG endpoint.
  28. Containment relations (S.r:A*B) are size inductive if its field is not child of its client. This is to allow methods to induce size into their endpoint but not into the class in which they are declared. Methods may be static and must still induce size into their client.
  29. Static rtFrom|rtPromise "F.r:F*F" or rtFrom|rtHas "F.r:F*A" relations do not induce size into the scope of their field.
  30. Changes to the Inflate algorithm to cater for methods. Setting the cardinality has changed.
  31. Center panel TLS of Tiger is 1 to not replicate the details already listed in Supra Editor.

Changes to the UML model to obtain re-compile-able code:

  1. The String, Bool, Date and Integer methods of INI_FILE had to be renamed with the RW prefix to avoid their clash with classes of the same name during model loading and merging operations.
  2. Certain methods of DICTIONARY and SCHOOL use double indirection to the SReference structure which needed to be modeled. Added the PSReference type to the model to reduce the double indirection to simple indirection.
  3. The DATE_TIME nested structure of TOKEN is under re-evaluation and was hidden away from the system's interface. All methods using it are taking and returning int instead.
  4. Removed tgHTMLComment and tgop as they were not in the HTML specification. Renamed the ! tag to DTD (document type definition) so as to be able to generate correct C++ source-code of the HTML category.
  5. Renamed the SQL statement to lowercase to not clash with the name of its category in the generated C++ code.
  6. STREE had two INIT methods that needed to be disambiguated. One is now INIT_GRAMMAR.
  7. Reordered declarations of some primitive types in the model file of BASE so that forward declarations are unnecessary in the C++ code generated by ANNA.
  8. Modeled the A,B relations of rect.
  9. Corrections to the formats of the core: revised the static methods and flagged them as such; added methods with ellipsis; TranslatedMoveBy had a problem with its line parameter; Methods must not be flagged as Abstract so that they map the .CODESEG type in the Type View; the self-relations of CAT_ABSTRACT were wrong.
  10. Removed one set of the LINE::GetNext and Line::GetPrev methods to avoid the clash in ANNA's namespace.
  11. CBMemScan callback is transmitted by reference to avoid a difficult to reconcile forward declaration. TYPE::SMembrane was hidden from the interface for a similar reason.

The new Compiler:

  1. Currently implements only the code generation direction.
  2. Generates code for every node whose visibility is not set to be at implementation-level.
  3. Generates enums for ANNA's internal scalar types.
  4. Types flagged as fleCategory are decompiled as both namespace and class under the C++ protocol. Category is a notion specific to ANNA's semantic core.
  5. NODE::Compile may be called with the cpCollectInterface or cpCollectForwards phase to either count or build the imports and forwards of a scope. Forwards is a list of objects whose use precedes their declaration. The forward relationship between two tree nodes is calculated by the new bool IsBefore(TREE*umbrella,TREE*l,TREE*r) function which compares the list positions of two children of the same parent, or that of their parents. The list of forward declarations within a scope is built by evaluating the wiring of each node in the scope.
  6. Under C++, NODE::Compile must segregate the relations of a category into a namespace and a class. It does so by testing the relation's GetKind classifier. This kind of a discrimination should not be at NODE level, however.
  7. LINE::Compile is not called for top-down source-code generation, and it is used to collect forward declarations and build the interface of imports in the compiler's vobjects vector.
  8. TYPE, RELATION and OPERATOR overload the inherited NODE::Compile and format their output according to the language protocol.
  9. Gaius LawView has a new ComboBox to pick the compile/decompile language. Moved the core serializor logic in the TForm_LawView::Compile method where it takes two turns, either over a HTML_STREAM or the former EditorStream.

TODO:

  1. Delete the STREAM::Style relation, style is an argument explicitly passed through the Compiler.
  2. Must compare the old and the new source-code compilers and gradually reduce them.


323 Tuesday, August 31, 2010

Enhancing the tokens of Edict dictionaries to hold more statistic information.

  1. Added Entry and Exit to TOKEN. Entry is the distance of this token from the dictionary's beginning as all words encoded so far. Exit is the last occurrence of the token measured from the beginning. The Exit-Entry distance is the Span of the token. These values show the proximity of a word to the language historic kernel, and the extent of its usage. Distant words are probably also less used and may vanish sooner than others. The 2 new fields and the Span linearly dependent pseudo-field are sort criteria for the dictionary.
  2. Emitting the column headings into the CVS at the beginning of dictionary export.
  3. Truncating the CVS file.
  4. Added Inputs to TOKEN to count the predecessors of each word in the grammar. Inputs is incremented in the Encode method whenever a new link is created between two symbols and is decremented in Decode when remove is specified. Removing the links is done when duplicate texts are unwound from the dictionary. Dictionary GetInfo now goes through Decode instead of going through the local cloned STEP logic; this is to keep link manipulation in one place. The profile showing the TotalLinksIn is not decremented on unwind as it should.
  5. Nedded the Double_Type and Int64_Type macros.


324 Wednesday, September 08, 2010

Started work on the ALGEBRA category. Added CAT_ALGEBRA to the morphism core with the new Matrix and MatrixSpace types.

  1. ALGEBRA is the beginning of an expert system that specializes in Mathematics. Its primary goal is to model matrixes and operations on matrixes. Added new editors for Matrix and ALGEBRA. The ALGEBRA editor should be a matrix expression builder and equation solver.
  2. Matrix is the type for matrixes of any size. Since concrete matrixes have different rows and columns, the Matrix class remains abstract. MatrixSpace is a type generator for matrix sets with identical geometry. A MatrixSpace instance is named such as S3x4Real is a set of all matrixes of 3 rows and 4 columns of Real cells. A matrix is ultimately generated by the MatrixSpace with the appropriate geometry. Matrix is a NODE descendent as all free-floating variables in the system need to be track-able in the Schematic Editor.
  3. Matrix::Set winds up the values passed as an amorphous buffer onto the lines of the matrix in a translation-like process.
  4. Matrix::Characterize calculates and returns a flag field that summarizes the matrix as being symmetric, triangular, identity, null, diagonal, etc.
  5. Matrix::Transpose is an in-place transform which turns the NxM matrix into a MxN one. If the matrix is not square (M!=N) then the
    transposition matrix space is created/located and the matrix migrates from the NxM space to the new MxN space.
  6. Deleting the last matrix of a matrix set should delete the matrix set as well and deflate the MGM.mgm constant accordingly.
  7. Colum and Row labels are kept in the Information field of each matrix and follow the transposition algorithm. LoadMatrix has the ability to load row and column labels if so instructed.
  8. Matrix::Resize is a difficult algorithm to redefine the geometry of a matrix and re-lay its cell data. The algorithm triggers metamorphosis and relocations. Cell data is moved at bit level with the bit-serial methods of OBJECT so as to be independent of the pivot type. Memory relocation may trigger, so caller must reload all pointers from the original address to the new address of the matrix which the method returns.
  9. Persist is called from the editor to serialize the matrix in or out.
  10. FindCreateConstant locates or generates named values of a given type. If a numeric matrix contains exceptional text entries, then alias constants are created and added to the constants of the numeric type which pivots the matrix space and the labels in the matrix are replaced by the constants. Matrix spaces that have one row or one column are aliased onto the line variant and are called vectors.
  11. ALGEBRA is the beginning of an expert system that specializes in Mathematics. Its primary goal is to model matrixes and operations on matrixes. Matrix sets are indexed into the MGM.mgm constant. The MGM is shape-shifting to accommodate all matrix sets in the
    system at a given time.
  12. The destructor of a morphic category destroys the meta information by apoptosis.
  13. The constructor of the ALGEBRA category creates several type masters. Real is a descendent from the nuclear Double to be referenced locally by matrixes. Matrix is a NODE descendent as all free-floating variables in the system need to be track-able in the
    Schematic Editor. MatrixSpace is the generator of matrix types of a certain geometry. Relations ~(Transposition) and * (multiplication) are generators of operations between compatible matrix spaces. The ~ and * relations form a graph of types compatible with respect to those operations. Matrix sets are themselves organized in a matrix of pointers to types. There's a special matrix type in the ALGEBRA called the Matrix Generators Matrix (MGM).
  14. The default action of the ALGEBRA category is to create several test matrixes and transpose them to exercise the metamorphosis algorithm involved in generating and registering new matrix spaces.
  15. Export virtual method to construct a matrix and link it to its matrix space. The matrix space for the specified geometry is located or created by an embedded call to FindCreateMS.
  16. LoadMatrix is a text processor that loads a matrix from a Comma Separated Value (CSV) file. Its flags indicate whether to interpret the first row of the CSV as column names and whether the first row is to be taken as the row names. Colum and Row labels are kept in the Information field of each matrix and follow the transposition algorithm. LoadMatrix is the parser, it creates the matrix after determining its geometry and its cell type, but invokes the core's MatrixGetSetCell to parse and set each cell value. If a numeric matrix contains exceptional text entries, then alias constants are created and added to the constants of the numeric type which pivots the matrix space and the labels in the matrix are replaced by the constants.
  17. FindCreateMS is to create a matrix space (set) or locate an existing one. The matrix space will be used to further generate a
    concrete matrix. The matrix space is located by indexing into the MGM.mgm constant.
  18. Matrix sets are themselves organized in a matrix of pointers to types. There's a special matrix type in the ALGEBRA called the
    Matrix Generators Matrix (MGM). This type instantiates a metamorphic constant named ,mgm which has as many rows and columns as encountered in the matrix sets. A matrix set of N rows by M columns is referenced by the .mgm's cell at position (N,M). The transposition space of an NxM matrix may be dereferenced in the mgm at (M,N). As matrixes are constructed in the system, so are
    the matrix sets that classify them. As new matrix sets are added, the MGM type gains more columns and rows and its .mgm constant grows.

Corrections to the core:

  1. TYPE::Inject is now allowed to change the instances of any type not just NODE providing that their names start with a dot. The Core is not yet ready to always relocate each and every instance. This is in support of resizing matrixes in our new ALGEBRA category.
  2. Corrected TYPE::FindRelation to use the modern GetFirstRelation, GetPrev pair to properly iterate a type's relations and locate one by name.
  3. Correcting a swag of new (and old) core destruction problems.
  4. MORPH has to explicitly delete the CODESEG instances and relation just after destroying the methods.
  5. The last step in the destruction of MORPH is to recurse to delete its inheritance relation. Memory locking and unlocking in this context would not do anything. There's code to trap interlocking closure problems which tests the kernel pointer. Protections had to be placed in the Lock/Unlock code against a null CAT_MORPH_Type pointer.
  6. Corrections to the destruction logic of the ALGEBRA category. CAT_ALGEBRA had unmapped relations which crashed the core upon destruction. The operators were not indexed. The mgm constant has to be destroyed explicitly.
  7. Found a major source of trouble in the GUI. The editors are not registered with the top EnterpriseMultiRelation and consequently their inter-frame correlation structures do not get deleted when closing the main frame of ANNA. These editors continued to be proxies to their target objects even after death, when the GUI was down. These phantom proxies provoked crashes in the core's proxy notification upward bubbling logic. Added a new vector to the EnterpriseMultiRelation to hold all these object editors. The Edit_OBJECT ancestor of all editors registers itself in that vector so that the destruction of the multi-relation would also destroy all such editors. To that end, there's a new virtual SetEndpoint channel in the TNullRelation hierarchy.
  8. Fixed the wrong dotrace flag passed from MemGrow to DoRelocation.
  9. Work on getting the core to relocate and also destroy correctly after adding the ALGEBRA category.
  10. Matrix had an unnecessary, unmapped field.
  11. Added the relocation methods to ALGEBRA and MatrixSpace.
  12. Added the Mgm bond to get to the matrix generator matrix directly.
  13. An enumerator for the cable between MORPH and ABSTRACT.
  14. Swapped the order of Matrix and MatrixSpace.

Found and fixed a treacherous bug in the relocator: DoRelocation called for objects whose header size differs from their block size, as it's the case with a matrix, must pass the starting offset at bit-granularity rather than byte. The logic of DoRelocation is that we call the system inbound type to relocate the first portion of the object, then call the user-defined type that extends the system type to relocate the tail of the object. The two relocations must be windowed.


325 Sunday, September 12, 2010

More work on the ALGEBRA category. Established the ring0 matrix functionality with the following features:

  1. Assignment operator = is useful in resetting matrix data.
  2. Equality test operator == compares the raw data of two matrices. Weak comparison ignoring matrix geometry.
  3. Inequality test operator != compares the matrix geometry and their raw data. Strong test.
  4. Matrix transpose is coded behind the ~= operator.
  5. The Rank calculation is currently trivial and returns min(row,col). The rank calculation is hidden behind the int cast operator.
  6. The generic matrix multiplier behind Matrix& operator ()(Matrix*A,Matrix*B) returns *this=A*B. Sensing for row or column overlapping so that the user may call it for in-place multiplications of square matrices.
  7. The most generic updater behind Matrix& operator ()(double p, Matrix*A, Matrix*B, double p, Matrix*C) returning this=pAB+qC. Operands may overlap.
  8. Operator + to add two matrices of the same size.
  9. An operator to calculate this=pA+rB.
  10. Right-hand-side square matrix multiplication behind the *= operator returns this=this*that.
  11. Left-hand-side square matrix multiplication behind the %= operator returning this=that*this;
  12. Operator (int row,int col) returning a void pointer to the matrix cell.
  13. Operator double (int row, int col, double type) returning the value of a matrix cell.
  14. These operators index operators have a significant overhead and cannot be used in performance loops.
  15. Matrix* operator () (Matrix* sub, int row, int col, void*& pcell) creates or fills and returns the sub-matrix by eliminating the line and column of the specified pivot. Also returns the pointer to the pivot cell. The operator is used in the computation of the determinant.
  16. Matrix cast to double calculates the determinant. Currently implemented algorithm is the Laplace Expansion. Algorithm has a fast core for ranks less than 10, and a slower wrapper where temporary matrices are created and tracked on the system's heap. Computing det(A) ehere rank(A)>11 currently seems to be prohibitive for my development machine. Time complexity is O(n!).
  17. Matrix::GetAlgebra is a direct and fast way to get the ALGEBRA category of matrices starting from a matrix pointer.
  18. ALGEBRA::Tests was renamed from DefaultAction and was linked to the system's test console. Whan called with -1, the method runs all other matrix tests until one fails, or until the end is reached.
  19. Added to the console a test of multiplications using various operators with result checking.
  20. Added a test for calculating the determinant of several 3x3 and 4x4 matrices with result validation.
  21. ALGEBRA::SaveMatrix is new and saves a matrix to a comma separated value (CSV) file. Fixed problems in LoadMatrix.
  22. Auto-naming matrix spaces.

A major effort to develop the matrix expression builder in the editor of the ALGEBRA category. In the left hand-side panel of the editor the user may pick from drop-down lists and build an evaluation tree structure of operators and variables-matrix. The expression is made out of instances of MatrixSpaceSelector or GrammarNode which is the base class. These grammar constructs are used in context sensitivity calculations along the derivation tree. The grammar nodes issue certain events such as Click, ComboBox_Change and ComboBox_DropDown to load and pick into the lists. Instance selection and operations are mixed into the list.

The matrix space and matrix selector combo boxes are now correlated for multiplication so as to select compatible matrices. Marking operations as unavailable for non-square matrices. Button::GoClick has a comprehensive list of matrix commands. A thread is spawned for determinant calculation.


326 Monday, September 13, 2010

More work on the ALGEBRA category. Calculating the inverse of a matrix.

  1. There's a new matrix method to calculate the Adjugated matrix of a given matrix. The Adjugated matrix is a "pre-inverse" matrix, whose cells are the cofactors of the initial matrix. Cofactors are the determinants of the matrix minors sign-swapped with the (-1)**(i+j). A matrix need not be invertible to have an adjugated form. The inverse is Adj(A) scaled by 1/det(A).
  2. The adjugation algorithm uses the Laplace expansion by minors the same as for the determinant.
  3. A new ! operator. Negation of a matrix is its inverse. The ! operator does the Laplace expansion on the stack. The danger in this function is not so much with the n^2+(n-1)^2+...+4+1 = n(n+1)(2n+1)/6 stack consumption, as is with the O(n!) time expense of the algorithm which makes it unusable for any matrix larger that 16.
  4. The ^ operator is out-of-place and creates the inverse from another source matrix. It calls internally the Adjugate method which uses the stack for small matrices and the heap for larger ones. Its use of the heap is not to do with resources as is to do with the ability to track these long living temporary matrices in the Scope Viewer and other such tools.
  5. Added a new set of matrix adjugation and inversion tests.
  6. Changed the MatrixTest function to use a tolerance epsilon of 1E-15. The comparison in ALGEBRA's self tests were failing due to precision loss.
  7. Added the new Get method to Matrix to complement the Set.
  8. Added the new Matrix::Filter(epsilon) method to clean-up a matrix of near-zero values.


327 Friday, September 17, 2010

Added the GEOMETRY category to the source pool of ANNA. The geometry module holds descriptive analytics for a growing family of shapes. Its minimal vocabulary is {Shape, Cloud, Sphere, Scene} The main objectives of the Geometry module should be:

  1. Construct, track, and destroy objects of the defined types.
  2. Define the interiority test for each type of object.
  3. Define union/intersection of certain objects.

The ALGEBRA has produced the Covariance and the Correlation matrices of a dataset.

  1. Found an error in our_strchr where the 0 at the end of the match string was considered a valid comparison term.
  2. Set of matrixes renamed to Group.
  3. Added the new SelectCols and SelectRows to Matrix.
  4. Matrix::SelectCols is a matrix convertor. It takes a string encoded filter expression such as 1:7,11==3,0==0M may be, which commands the creation of a new matrix with columns 1 to 7 copied from the original, only those rows that have a value of 3 in the 11th column of the original, where any value of 0 in any cell is to be replaced by the mean across the column. The filter produces a reduced matrix as well as 3 new nested others with the following significance: summary, covariance, correlation.
  5. the summary matrix with each row describing the column of the reduced parent matrix. First column of summary is for the mean values; second column is for the standard deviation, third column is the counter of null values.
  6. The filter produces the covariance square matrix as child of the original.
  7. The filter produces the correlation matrix which reflects the linear relationship between variables of the original dataset.
  8. Renamed MatrixSpace to MatrixGroup.


327 Monday, September 20, 2010

Corrected and enhanced the ALGEBRA category

  1. TYPE::GetInstanceByName was enhanced with the scope argument which, when it is not null, is used to select a type's instances that are also in a scope.
  2. Matrix::SelectCols is not creating the Summary, Covariance and the Correlation matrices. That has become the job of 3 new dedicated methods with the respective names.
  3. Fund and fixed 2 problems relating to the mean substitution in SelectCols: mean substitution was not done for all columns that had 0-es; the mean was calculated after the filter was applied and should have been applied on the entire column of data.
  4. Added args to Summary, Covariance and Correlation. This parameter may be parsed for further details such as the SS/N or SS/(N-1) analysis model. A null args value defaults to SS/(N-1).
  5. The Summary, Correlation and Covariance matrices are now children of their pivot dataset.
  6. The new Matrix::Eigenvalues method calculates the eigenvectors and eigenvalues using a convergent Jacoby algorithm. Two new matrices are created and nested inside the target square and symmetric matrix: the eigenvectors matrix and the eigenvalues with 4 columns: value, percent from total variance, cumulative eigenvalue, and cumulative percent. The method directly returns the eigenvalues matrix.
  7. Added a console test for eigenvectors.
  8. Has to give-up the aliasing of one-dimensional horizontal and vertical matrices and keep them into their distinct groups. Certain multiplications/transposition operations were failing.
  9. Found and fixed a bug in matrix operator +.
  10. Added the chart display for the scree-plot of eigenvalues.


328 Wednesday, September 29, 2010

Added the GEOMETRY category

  1. Added the GEOMETRY category to the morphic core.
  2. Added the new CANVAS3D to the STREAMS category.
  3. Constructing a system-wide 3D canvas as the premise to rendering the schematic in 3D.
  4. AutoMerge GEOMETRY and ALGEBRA.
  5. Corrected a problem with null names in NODE::Merge.
  6. Enhanced the POINT3D branch.
  7. Found and fixed a mapping problem in the formats of ALGBRA.
  8. Added the editor for a CANVAS3D.
  9. Changed the Schematic Editor to advance the directX scene only when sliding the scrollbars.

Integrated a modified version of Microsoft's DXUT core into our KBASE library.

  1. Changed so that the presentation of the scene is done on a specified window.
  2. Wrapped DXUT in our DXD9WRAPPER class with a few methods for the new STREAMS::CANVAS3D.
  3. A swag of upgrades to our SDK directory to be able to build DXUT samples.
  4. Heavily reduced DXUT from something like 40,000lines of code and 20-odd files to a single file of 12,000 lines of code that still renders the ShadowMap scene.


329 Thursday, September 30, 2010

The objective of this phase of development is to "appropriate" the ShadowMap Direct3D sample from Microsoft and tailor it so that the scene is a STREAM and it may be instantiated multiple times in the same process. To this end we had to control the global variables into a DXUT_WRAPPER class, include an instance of the wrapper into the CANVAS3D stream, and re-work the callback mechanism of the sample to be virtual method driven. The methods of DxD9 cannot be virtual.


330 Friday, October 08, 2010

More work on UML 3D.

  1. The default constructor of CANVAS3D creates sub-objects, unlike any other type in the system. To extract the VMT we had to introduce another BIND_DNA0 macro to create a stack variable which would be destroyed on exit.
  2. STREAM::RotoPoly is a new method to render arrow-heads. CANVAS3D overrides this method to render a cone.
  3. Added depth and open arguments to DxD9 Polygon. Added new methods to DxD9 class: RotoPoly to render cones; Ellipse to render spheres, Line to render cylinders as 3D extensions of the 2D wires.
  4. Added a destructor to DxD9.
  5. Reworked the DxD9 responsibilities over several new methods: ResetScene, RenderScene, ChangeCoordinates. Render scene is a necessary callback to the CANVAS3D which must serialize the ANNA K5 structure.
  6. Turned DxD9 from UNICODE to ANSI.
  7. X definitions and textures are now in the resources of the monolithic application or in those of BASE.DLL.
  8. Reduced the size of the FileMesh class by removing the 1KB strName field. The creation of meshes is a tremendously memory consuming operation.
  9. WrapperRender and ClearDynaMesh are new wrapper functions to allow the scene to be reloaded/replaced.
  10. Extended the CGrowableArray::SetSizeInternal to also shrink the array.
  11. There's a crash in the destruction of the object array which we're currently hunting down.
  12. Under Borland there's no default == operator for class instances, we're using a memcmp instead.
  13. Reduced the scene to a minimum. We kept the room with its walls, the light, the sphere and the airplane. The viewpoint can now be attached to the airplane along the centripete direction.
  14. All 3D mesh objects are now allocated in a growing array.
  15. The DxD9::Polygon method takes a K5 polygon description which is a sequence of points that in 2D would define a closed path and extends a 3D mesh from it. The method creates a vertex list, a list of triangle faces, and an adjacency list as an extension of the 2D polygon to a 3D polyhedron.
  16. The new DxD9::Ellipse creates a sphere at origin, then positions it to the specified space position.
  17. The new DxD9::Line method creates a cylinder at the origin which it then roto-translates into the 3D position.
  18. The new The DxD9::RotoPoly creates a cone at the origin, then roto-translates it into the proper position in space.
  19. Work on the 3D viewer to allow the reload/replace of structures into the scene. DLS and TLS buttons. Refresh timer enable/disable button. Viewpoint change button.
  20. CANVAS3D was extended with RotoPoly, Ellipse, DisplayScene, and SceneAttach methods. The CANVAS3D tracks its scene via its Scene BOND. SceneAttach inserts the canvas ito the target node's Proxies bond.
  21. CANVAS3D::Polygon, RotoPoly, Ellipse, and LineTo reverse the Y orientation of the 2D structure . RotoPoly scales the structure by 100 to try to avoid errors in rounding operations.


331 Saturday, November 27, 2010

Back to working on ANNA after the development of the NANO MetaCore.
NANO is a rework of ANNA's proto-nucleus and would eventually overtake OBJECT__BASE, OBJECT, TREE, and would unify NODE, LINE, TYPE, RELATION, and THREAD.

A series of extensions to the Core and GUI to bring back the Instrument Server evaluator in the same state as it was 10 years ago.

  1. Renamed the flags to index in TYPE::GetInstanceName and enhanced the logic to match a type instance by name, scope and index.
  2. A protection against moving kernel owned nodes to other namespaces in TREE::Relate during drag&drop.
  3. NODE::RuntimeAttach was temporarily brought back into the model. The modern thinking, however, is to use the serializor to collect the GUI of an instrument.
  4. RuntimeAttach binds the node to a NotifyRelation object.
  5. Added NewTrace and NewCheckBox to TNotifyRelation as virtual methods for NODE::RuntimeAttach.
  6. Added IncTick and GetTick to the core to be used by instrument evaluators as time base.
  7. Added the flEvaluating flag to LINE.
  8. Added the arithmetic component to InstServ.
  9. Gone back to the old and primitive evaluator to get the basic functionality back.
  10. Double-click on a node loads the instrument into the Runtime window via LoadObject.
  11. Added CheckBox to the UI.
  12. Needed a way to make constants. Enhanced the Schematic Editor with an action to toggle a node between selector and constant.
  13. Tried to update the serializor. Some crash-protections in the Scope Viewer to do with closing streams with sub-locks.
  14. Fixed some wrong ClassID bindings in category SetTypeMaster.
  15. Enhanced InstServ to the level of year 2000. Added the Arithmetic instrument to the model. IHost descends from NODE rather than TYPE.
  16. Added the evaluation logic to IArithmetic.
  17. Clearing the parent of Edit_NodeData and of Edit_NodeMaxSize lead to crashes in the Schematic Editor. We're constructing the SE with these editors set to invisible.

A number of changes to fix the Serialize algorithm:

  1. Found that the relative path formation and FindCreate were wrong in calculating the duplication ID of a node in advance of the type and the name filters. The logic failed when objects of types other than the one we're locating are inserted in the scope above, which is a common scenario. Changed the loops in GetDuplicateID and FindCreate. There was a general problem in FindCreate in that the required index of the duplicate was ignored. Fixed at the expense of having to regress-test the entire system.
  2. A fix to FindCreate unnamed objects.
  3. Avoiding malloc on names in FindCreate.
  4. IsSameType in object.cpp should return true only if the filter is set to ANY_TYPE.
  5. The recent Ins bond of TREE was not serialized.
  6. Object ownership has been generalized so that all objects have a container. Serializing must not traverse to the container if the object is flagged as disowned.
  7. Added the noname separator to the serializor's PathSeparator string.
  8. We now leave the autobind on during serialization so that wires are automatically bound to cables.
  9. A protection in LINE::GetName against null From endpoint.
  10. Changed STREAM::Nest to filter stream types. This change might be too restrictive, though and needs to be verified.
  11. Scope View now checks if the node is locked and avoids to save it if it is. Saving locked nodes lead to corrupt files.
  12. More protections against tree anomalies.
  13. There's much trouble serializing cables that cross an object's IO boundary.
  14. NODE::IsSelector had to change as all objects are now contained.
  15. The Shape combo of selectors had to be filled with trace shapes.
  16. NODE::RuntimeAttach opens runtime windows for all selectors.
  17. Cannot specialize STREAM in the UI to make a TRuntimeStream. It's due to the VMT that must not link the core in.
  18. Updated the Generator form. Added the generator UI to the Navigator.
  19. Blended in the capabilities of MORPH.EVL. Not working yet, but mounting the runtime traces for a morphologically defined instrument.
  20. Traces now use the special fields of IHost to hold screen position info thus leaving the schematic unchanged.
  21. Drag and drop in the schematic sets the position of the created node.


332 Friday, December 31, 2010 (Evaluator, HMM milestones)

Added the HMM logic as Physiology Graphs

  1. Added the PHYSIOLOGY type to the ABSTRACT category to build Hidden Markov Models for signal generation and signal recognition using the Physiology Graphs formerly built by the GUI. The justification, design and workings of the Physiology Graph is described in the "Physiology" and "HMM Generator" whitepapers. PHYSIOLOGY is a descendent of PORTAL and a memory sub-allocator.
  2. Enhanced the Evaluate method with the time argument used in recursive evaluation to evaluate an algorithm only once per pulse. The new evaluator uses an object's last modified time field to synchronize with the evaluation time argument.
  3. Added NODE::flCancelled flag used in Physiology evaluation.
  4. Reversed the traversal direction of NODE::GetLine.
  5. A fix to NODE::handle_realloc to avoid changing the MaxSize of a constant that doesn't have a handle.
  6. All new nodes are born out of TYPE::CreateInstance as constants without handle.
  7. Preventing lines from rendering when their endpoints are above TLS.
  8. Some fixes to the line drawing algorithm. Better rendering of wires with endpoints in ellipse shaped nodes. Rendering the cardinality at the from endpoint of a relation.
  9. Fixes to OBJECT::UserInt and UserDouble.
  10. Needed both possible solutions of the point::LineAndEllipse intersector.
  11. Modeled a few businesses such as the Employment Agency and the Transport Co.
  12. Enhancements to the ALGEBRA category.
  13. Matrix is now an instrument that can be evaluated. Matrix evaluation is to evaluate all input lines, shift the matrix up by one row, add a new row with the input values as the last row of the matrix. One may now watch a matrix as a plot of incoming values.
  14. Enhanced the Matrix::RandomFill method to produce stochastic matrices and HMMs.
  15. Added Matrix::BuildGraph.
  16. Matrix overrides the SerializeChildren method. A new STREAM generation style UI. A matrix may be serialized to a canvas as a plot.
  17. Enhanced the CAT_ALGEBRA::Tests with HMM constructs. The main is the Viterbi Baum-Welch construct.
  18. Matrix is born as a constant.
  19. Matrix::BuildGraph creates normal graphs or physiolog graphs as instructed with the textual argument.
  20. Added Matrix::NeuralNetwork to create a hidden-layer neural network.
  21. An attempt to set the formats of the instrument server category.
  22. Enhanced the CAT_MORPH::CreateLine wiring tool to allow wiring between selectors and relations.
  23. Avoiding hiding and showing the Edit_NodeData and Edit_NodeSize controls as window handles cannot be destroyed from a thread other than the creator.
  24. TForm_Relation::PleaseDoThatClose calls the Close method instead of Release so that FormClose handlers are being called.
  25. Renamed Physiology GUI to Phys, as there's now a core type called Physiology.
  26. Changed the Phys viewer to work on different styles of allocators: thread, portal, or physiology. We're working to reduce this component and generalize it to the new editor of the core PHYSIOLOGY graph.
  27. Matrix editor is now enhanced to show the matrix as a chart or as a core-rendered plot.
  28. Gauge timer in EditFrame seems to be a problem. Constructing in disabled state.
  29. ALGEBRA editor was enhanced with more parameters for RandomFill, and BuildGraph. Added more integrity tests for the category.
  30. Fixes to allow evaluation of instruments in both Enterprise and Morphic models.


333 Sunday, January 24, 2011 (ERP milestone)

Enhancements to the core to start-off the ERP project. Designed the new ERP category with the following types: PROJECT, PERSON, PLACE, ACTIVITY, SKILL, PRODUCT, JOBSPEC, RESUME, PLAN. The justification, design and workings of the ERP is described in the "Fuzzy ERP" whitepaper. The ERP category currently has a type editor for the PROJECT type. This is the viewer/editor of the bipartite graph between the ERP objects.

FindCreateUML must do IsSameType with the supplied filter rather than FIND_EXACT_TYPE. Category merging fails otherwise. FindCreate must do the same so that the XSD schema may be browsed.

More fixes to the STREAM so that the XML parsers work. Implemented the MEMORY_STREAM::IsEnd.

ALGEBRA Relations must be the first field before Mgm, just as is the case with any other category for proper relation mapping.

Test results for V333

  1. UML loading okay.
  2. UML AutoMerge leaves an incompletely mapped UMLCODESEG.
  3. Can serialize the entire MORPH category out and back in.
  4. Crashes in the memory relocator during manual defragmentation if AutoTIV is enabled during UML import and more than the ABSTRACT category are merged.
  5. Memory Viewer loses hold onto the defragmenting allocator and needs to be closed and reopened.
  6. No code text for any merged category other than ABSTRACT. May be due to deleting the UMLCODESEG.
  7. Physiology Graphs working.
  8. Dictionary creation still working. Monitoring them in the Physiology graph is okay.
  9. Exits cleanly. The .code constant, however must be destroyed before the methods, so that deflation isn't triggered during system shutdown.


334 Sunday, March 20, 2011

Work on connecting ANNA's portals to a MSSQL server via an ODBC connector.

  1. Got FSYS::DATABASE to work again.
  2. Added checkpoints for the HOST_OS_SQL bridge.
  3. Publishing the LocateTableByName from CoreAPI to support the loading of tables from the new PORTAL::MountDatabase logic.
  4. Enhanced TYPE::FindRelation with new arguments to locate by name and also by flags, cardinality and supplier, if specified.
  5. Found a problem with the depth of serialization of lines.
  6. Relaxed a test in OBJECT::Unlock for running certain algorithms such as SQL DB mounting from console when no UI threads are available.
  7. Enhanced PORTAL with the SQL access methods borrowed from FSYS::DATABASE. The new methods are: Seek, Create, CreateTable, DeleteTable, Alter, SqlExec,
  8. New helper methods: PORTAL::RecordWrite, RecordRead, RecordAdd and RecordRemove.
  9. Added PORTAL::DbTest suite which currently calls MountDatabase.
  10. Changed OTP to not emit the INI file.

335 Sunday, March 06, 2011 (MSSQL DB Loader)

Work on connecting ANNA's portals to a MSSQL server via an ODBC connector.

  1. Added checkpoints for the HOST_OS_SQL bridge.
  2. Proxima's logo on the back wall of Canvas3D.
  3. An attempt to circumvent the problem with D3DXCreateEffectFromResourceA.
  4. Reduced the DirectX resources of ANNA.
  5. Added a test button on PORTAL editor to launch the DbTest suite. It currntly mounts a database specified by the ODBC DSN with the same name as the portal.
  6. Found and fixed a problem with self linking a relation in RELATION::INIT.
  7. JustBlanks moved from HTML to PORTAL.
  8. The PORTAL emits method bodies onto the web. This may expose ANNA's code. (TODO!)
  9. PORTAL::MountDatabase does a series of queries onto the connector to get base types, tables, columns, associations, procedures and parameters.


336 Sunday, March 20, 2011 (Type Category Reductor)

A new algorithm to reduce a type category.

  1. Enhanced TYPE::CompareTypes and CompareReduce methods. CompareReduce can now factor common relations to intermediary types. This is an important result.
  2. Added the TYPE::Reduce method that looks for empty intermediary types and re-links their descendents to bypass the empty type. The algorithm finally deletes the empty orphan types.
  3. A new TYPE::Sort mechanism that looks for dependents and emits them first.
  4. Added another flTag flag to NODE. The tag bit is used to mark the passage of recursive algorithms such as Sort .
  5. Enhanced the Schematic Editor to do promotion on a multiple selection. Added action handlers for CompareReduce, ReduceEmpty, and Sort.


337 Saturday, April 30, 2011

Fixes and enhancements.

  1. LINE flags renamed from va to fl.
  2. Changed the PORTAL's MountDatabase method to not call DoCompareReduce on the database category. That process is now available on the TYPE menu as the Compare Reduce option.
  3. Added the new GroupRelations method which is meant to do the transformation from class category to sentences.
  4. Enhanced the type map formatter to convert date and time fields to string and show them in Type Viewer.
  5. Went back to point 2D as the 3D matrix transforms are not ready yet.
  6. LoadResourceData must not issue log messages as a race condition arises.
  7. FormatDateTime is new in HOST_OS.CPP
  8. Added a CONTRACT relation to the ERP category. JOBSPEC and RESUME descend from it.
  9. Placed our company logo into the 3D world on the back wall.
  10. A new parameter to CanvasStream to fit the picture into the limits of the canvas or to maintain the aspect ratio.
  11. The low-level CanvasStream is now capable of rendering RGB colors as well as indexed, depending on whether the ColourTable is present or not.
  12. Greatly enhanced our diagram printing capabilities. We now render on print devices at any resolution.

Implemented the Report Generator

  1. There's a new LINE::flDocument flag that drives the Report Generator instead of line drawing onto the schematic. Reports are generated at 300 dpi.
  2. Implemented the Report Generation logic of DGRS into ANNA's Schematic Editor.
  3. A new LINE::SerializeDocument method is entered for lines that have the new flDocument flag set. The method renders a first degree header and footer with logo bitmaps, serial number, page/pages in barcode. It renders a second degree header with details of a line's endpoints, verb, and complement as well as a second degree footer with details and place holders for the signatures of sender, recipient and author.
  4. New GetFieldName and GetRegion methods to bring labels, values, and geometry back to SerializeDocument. A new enumerator hierarchy for objects to identify their fields for the report. The report generator renders the content of a cable as tables and trees of details in such a way so that OCR is possible over the resulting image.
  5. Implemented the time bubbling algorithm of OTP and NANO into ANNA's BASE core. Renamed CreationTime to TimeStart and LastModified to TimeEnd. Base object now implements the Touch method. All items show their dates onto the report.
  6. Added Page and Pages to the STREAM.
  7. Added left and right oriented logos of Proxima Centauri to Tiger.

DICTIONARY enhancements

  1. Added the DICTIONARY::ForWordContext method to build a list of closures of a string of words selected in a dictionary archive. It calls the new WordLocateByName to get the handles of the words in the test sentence for which it builds a bi-partite graph of possible sentences that include the test. The method then calls the new DICTIONARY::MatchEncoding method to verify which of these paths are valid in the dictionary by deconvolving the history of the archive for patterns. The valid targets are tagged into the index as results of the search. An estimator function and a rank would then need to be built. MatchEncodings is a decoder that scans the archive from the first occurrence of the first word in the sequence until the sequence is matched or until all hit-counts were exhausted during the test. The decoder is an extension of strstr and has with a lower time complexity as it has late entry and early exit logic.
  2. Added the Rank field to the tokens of DICTIONARY. These are about to be used in disambiguation.
  3. Added a new DICTIONARY::GetPrev method to return a word's specified predecessor. Since a word's back-links cannot be used as they cannot yet be built (this is due to us not being able to fully revert the state of the archive after a duplicate text is found), the algorithm must back track over the index to test the forward links of the previous words.
  4. Changed ForWordContext to receive a string rather than a token.


338 Friday, May 13, 2011

Enhanced the DICTIONARY with the ForGraphPower algorithm that builds the list of most common sub-expressions found in the archive. There's a new UI control onto the Edit_DICTIONARY tab to configure the process and list the results as EDICT gathers them. Changes and corrections to MatchEncoding to have it work in two modes: first hit with returning the container text, or exhaustive search returning the number of occurrences. The sub-expression finding algorithm exposes its inner workings mostly onto the profile panel as chart data. There are new performance counters in the dictionary : ExpressionRoots, ExpressionTests, ExpressionTestsLocal, ExpressionHits, ExpressionFanning, MatchEncodings, MatchEncodingsFrom, MatchEncodingsRotor, MatchEncodingsTo. The NEW_LINK_WORDS is enabled, but still under testing. Added a profile action to bring the selected counter to zero.



339 Tuesday, January 17, 2011

Image processing.

  1. A Nx5 matrix can now be styled as image and thus be rendered on Canvas as pixels.
  2. The Algebra's Matrix Group of Matrixes is used for instant location of matrix types used in linear algebra calculations. MGM was a problem for large rows/columns. It is now limited to 64x64 and used as cache for instant location of compatible matrices. Any matrix with geometry smaller than 64 hits the cache, whereas locating larger matrix groups is done by linear search in the Algebra category.
  3. There was a bug in setting the cell value.
  4. There was crash condition in the Covariance routine.
  5. Integrated LIBTIFF into the Anna's kernel. Changes to build under Anna's build model. Bound to Anna's memory manager. Added a parameter to RenderTiffFile to specify the BOTLEFT orientation. Got contention on the canvas of tree view controls.
  6. Enhanced the image processing pathways of Anna with a direction argument that now also enables reading images. A new image type is TIFF. Low level routines to Get/SetPixel onto device contexts.Found problems in the CreateitmapBuffer and Bmp2Buffer where the header had to be BITMAPFILEHEADER instead of BITMAPINFOHEADER and the offset to bits was wrong. There are changes to DrawTransparentBitmap. A new CopyBitmapBits function that reads a specified region of a device context into a memory resident bitmap file.
  7. ScopeView launches the external actions on a secondary thread.
  8. Enhanced the SchematicEditor with variance analysis of its contained image. If the new STREAM::flAnalyse flag is set, then SE is performing XY projection on every repaint. Canvas now has Get/SetPixel methods. There's a new Analyze canvas method to compute the image projections onto the XY axes. Stream editor sets the XY projection mode of canvas and exposes the Analysis button.
  9. Relaxed the palette constraints of displaying to Canvas. One can now draw with true RGB colors.
  10. Enhanced the matrix editor with virtual grid controls so that we may open large objects.
  11. There's a new toolbar in the object editor to work on image constants. The action button launches the image analysis module that returns the stabilization angle.
  12. NODE has a new GetSkewAngle method that wraps the new image processor.
  13. Angles throughout Anna have been redefined to the 10th of a degree for precision.
  14. New point operators for -=, *=, /=, *, +, -. /. Added the point3D class in case the core is built with 2D points. A new rect(bool) constructor to build a degenerate rectangle. A new rect3D class for RGB manipulation. The2D matrix is now using a sine table for speed.
  15. Images loaded by the abstract core are always nodes. New image nodes are of the size indicated by the image. CreationTime and LastModified were renamed to TimeStart and TimeEnd.
  16. We gave-up on automatically showing the 16x16 icon for nodes of height 16.

Work on the OCR module. Added the deskew logic.

NANO changed with all methods __cdecl so that the "this" pointer is passed explicitly onto the stack. An important step towards a MSVC built core with a Borland CBuilder user interface. The NANO service is now unmanaged.

DGRS. There was always a problem with the EXCEL output tab. Removed. Started on the compiler of ID cards. Added a button to clear unused products.

Added the maths directory to the source project. It contains scripts for deskew, viewers and tests.


340 Tuesday, January 31, 2011 (deskew)

Image processing is discussed in the "Preprocesor pentru compilatoare de imagine" whitepaper.

  1. A new debug trace flag DBG_NONE=0 that does nothing.
  2. HOST_OS_GDI::ImageDelete returns false if the image argument is null.
  3. Algebra is now using the new TYPE::flAutoDestroy flag to automatically delete the matrix type master when its last instance is destroyed.
  4. The binary classification trees of the new image classifier are much deeper than the limit of 10 prescribed by the anti-cycle logic of the interlock, and as deep as the number of nodes in the original graph. Relaxed the limit to MAXINT.
  5. Fixed an MGM problem during relocation. Matrix SerializeFace is now considering that RGB is all coded in one packed column. A color of -1 is foreground and a -2 is background.
  6. Added another NODE::ofRenderBackToFront option to reverse the order of showing the nodes in the schematic.
  7. New virtual node OpenImage and CloseImage methods.
  8. Added 2 new algorithms to the abstract core: Deskew and Ocr/Classification that describe the new imaging algorithms using physiology roulettes.
  9. A change in the file importer of the abstract core to not actually open the files when the flags argument has bit 0 set.
  10. Method NODE::GetSkewAngle now expects the image to be opened by the caller.
  11. There was a problem with the convex membranes that were immediately set by the TREE editor on show page.
  12. A protection in the Schematic Editor against bit fields with a larger Max than the count of checkboxes in the editor.
  13. Enhanced the image processor NODE::GetSkewAngle to be aware of object relocations. Enhanced the trace messages. Sharpness threshold from 10 to 12. Vector image is now 3x wide with RGB coded as one packed value. Debug modes to keep the estimation, convergence, and final image matrices as well as the tree of octants constructed by the algorithm that finds the dominant color.
  14. EditErp is a new UI with controls to: load the content of a folder, pass each file through the deskewer, construct a classification graph and gather the dissimilarity scree-plot, let the user select a number of classes, build the classifiers and do the classification, construct stock classifiers such as for identification cards. The logic is creating a neural network for auto-classifying images.

TODO:

  1. Move the classifier from EditErp to Core.
  2. Rename GetSkewAngle to Deskew.
  3. Update the UML to reflect this month's changes and regenerate the code from the model.
  4. Resolve the crash on shutdown.
  5. Speed-up matrix relocation.
  6. Reduce TOtpObject::Associates to TREE::Ins.

341 Sunday, February 19, 2012

Some fixes:

  1. CAT_ERP had to be reordered so as to have the PROJECT type master last. This is to correct the Core destruction problem when connectors were destroyed before the project, leading to cycles.
  2. CAT_MORPH::ptrGlobals, ptrFlags, ptrAutoload, ptrDebug, ptrPPrintMessage were not used and not initialized, leading to core destruction crashes.
  3. SerializeDial now uses angles in the tenth of a degree.
  4. Corrected the relation dialer to cope with auto-relations such as the Abstract core.
  5. An extension to NODE to load JPG files as BMP nodes. Jpg2BmpBuf, JpgInfo are new.
  6. Better debug flags for GetSkewAngle.
  7. Annother attempt at an algorithm to remove the jagged frame left on one of Anda's ID cards that has black oblique borders .

342 Thursday, February 23, 2012

NANO b42:

  1. An experiment to gain indirect access to the BOND pointers. Succeeded to have virtual POYC pointers through operator[].
  2. Ported to BORLAND BC502.
  3. Borland doesn't have __FUNCSIG__ used for trace.
  4. Borland runtime lib doesn't have findclose.
  5. The name field of struct SFork was in conflict with TASM.
  6. Implemented a double jump at the top of Fork to avoid multi-pass TASM.
  7. Directives to disable optimizations around Fork are different for BC502.
  8. Borland compiler doesn't implement __event and __hook intrinsics. The browser refresh timer was disabled for now.
  9. FONT declaration in the resource script has fewer arguments in Borland's resource compiler than MSVC v9.
  10. Borland doesn't allow us to call main from within the program.
  11. The NUI timer used for browser refresh is now a pair (g, timer) - (pointer to instance, pointer to method) rather than a MSVC speciffic __event. This was an important step towards building NANO in the same build structure as ANNA. The (g,timer) closure cannot be called from a static method, so GTimer::DoTimeout nou calls the new normal GTimer::Timeout method, which in turn calls the closure.
  12. Implemented _finddata_t, _findfirst, _findnext, _findclose for Borland to support browsing the file structure.
  13. Turned all methods to __cdecl in an attempt to level-out the mangling differences between Borland and MSVC.
  14. Since Fork is now __cdecl, the new this parameter is pushed onto the stack, hence SFork has a new data field.
  15. The same __asm statements for both Borland and MSVC. Resolved the long standing issue of BC502 generating and not clearing the obj.000 file which so far required BASM. We're using the script TASMFIX.SPP
  16. #pragma optimize are different from Borland to MSVC.
  17. Disabled the payload code of the service.

CORE:

  1. Built ZLIB with FASTEST enabled to increase the throughput of Anna's web server.
  2. Covered a series of memory relocation problems which now permit much smaller portals to be instantiated.
  3. A series of changes to PORTAL to handle memory relocations. The new logic was tested only on WEB portals. FTP and OTP must be revisited.
  4. The demo structure of a WEB client is now created within the client's allocator rather than the server's/
  5. The convex membrane of the legend triggered recursive metamorphosis of the allocators and was taken out.
  6. Proxy updates now check the protection bit and skip protected chains. One cannot use the proxy notification mechanism during relocations.
  7. STREE_BASE::GetTreeAspect is now calling the IsObject tester which is quite expensive.
  8. The FlatCount of the Proxies gene had to be initialised.
  9. Discovered a new technique of taking proxies to amorphous blocks by inserting the proxy as child of the used block. Memory manager has to relocate the parent pointers of such proxies. Relocation of such under-proxies is now done in TYPE::MemGrow which is not the proper place, should be done in Type::MemOperation at the point of moving a used block into a free block.

343 Wednesday, March 14, 2012

CORE:

  1. No GuiDispatch during HOST_OS_RT::FileAccess... again.
  2. Promote was wrong in that it inverted the order of the nodes.
  3. COMMENT tag is "!" and is a leaf.
  4. List item requires an end tag.
  5. ParseHtmlSection now takes a flags argument where bit 0 set is for the new algorithm. SCHOOL still uses the old one.
  6. File path doesn't need a fully qualified path. There was a bug when splitting the path to get the file name.
  7. The right limit for ParseHtmlSection was inexact.
  8. CollectInfo was called with a flag to collect in the back-to-front direction. Corrected the order.
  9. CAT_SCHOOL::TestOnEmbeddedData is new and used to extract the resource files used to test the Edict encoder.

344 Wednesday, April 04, 2012 (milestone EDICT-GAIUS working again)

CORE:

  1. Changed the activation logic to pass through the permissions bit field. Gaius Navigator can now be licensed in "developer" mode with bit 2 set, in which it allows access to word level.
  2. Changed the serializor to support loading and saving dictionaries. There's a new STREAM flBinary flag used for "enterprise" structures. TYPE serializes by first disabling its heap, storing/loading it and then re-activating the heap. The process involves memory relocation. TYPE::SerializeHeap had to be passed the pointer to the allocator, not just the limits of the heap, in order to update the type's TIV.
  3. The new version 14 of our file system serializes the MaxSize field of any node descendent including TYPE and DICTIONARY. The maxSize is now known in advance of loading and of CreateInstance in order to accommodate the heap.
  4. Version 4 of the dictionary serializor together w