All about the development of Exponential (CMS) by Graham Brookins, 7x (formerly Brookins Consulting) and our think tank kracker.org.
Hello Exponential Community,
7x keeps the complete doc.ez.no documentation as a static HTML archive in a public GitHub repository. Two free Exponential extensions turn that archive, or any folder of HTML or Markdown, or a live website, into real content objects in your own Exponential site with one command. One run gives finished work; running it again is never needed, but always safe. The result runs live at doc.exponential.earth.
The repositories
- se7enxweb/exponential: Exponential 6.0.15 or later, the CMS.
- se7enxweb/exponential.doc.exponential.earth: the static archive of doc.ez.no, 88,680 pages.
- se7enxweb/exp_static_html_imports: the import framework. Four source types, your own types, safe re-runs, reports.
- se7enxweb/exp_doc: documentation content classes, the import rules for the archive, and two designs: Classic (doc2025, the static site exactly) and Modern (doc2026).
All are free software under the GNU General Public License v2.0 (or any later version).
What you need
- Exponential 6.0.15 or later (earlier 6.0 versions work without the audit events and the console aliases).
- PHP 7.4 or later; PHP 8.0 to 8.5 are supported.
- Any of five databases: MySQL/MariaDB, PostgreSQL, SQLite, Oracle or MongoDB.
- The online editor ezoe, active by default. Its HTML parser turns pages into rich text.
- git on the server, for sources given as a repository.
- For exp_doc also: eztags (labels) and xrowmetadata (the metadata attribute).
Example 1: a folder of Markdown, in minutes
Install the framework from your Exponential root:
git clone https://github.com/se7enxweb/exp_static_html_imports.git extension/exp_static_html_imports
Activate it in settings/override/site.ini.append.php:
[ExtensionSettings] ActiveExtensions[]=exp_static_html_imports
Deploy, check, look first, then import:
php bin/php/ezpgenerateautoloads.php -e php bin/php/ezcache.php --clear-all php bin/php/console exp:static-import:types php bin/php/console exp:static-import:run --markdown --source=/path/to/docs --parent-node=2 --dry-run php bin/php/console exp:static-import:run --markdown --source=/path/to/docs --parent-node=2 php bin/php/console exp:static-import:status
- exp:static-import:types lists the four types: --doc-html, --html-generic, --markdown, --sitemap-crawl.
- --dry-run counts what would be created, updated, flagged and removed, writes a report, and changes nothing.
- Each Markdown file (.md, .markdown) becomes a Folder below node 2: name from the title, short description from the body. README.md and index.md are the index page of their folder. Front matter is read; code blocks keep their language.
- On Velocity, php bin/php/console exp:velocity deploy --allow-root-user replaces the two deploy lines.
Example 2: a folder of HTML, or a live website
php bin/php/console exp:static-import:run --html-generic --source=/path/to/html --parent-node=2 --dry-run php bin/php/console exp:static-import:run --html-generic --source=/path/to/html --parent-node=2 php bin/php/console exp:static-import:run --sitemap-crawl --source=https://www.example.com --parent-node=2
- --html-generic reads .html and .htm files; index.html is the folder page. The title comes from h1, the body from main, article or body; nav, header and footer are removed; code in pre keeps its language. Linked .pdf and .zip files become File objects, other assets Image objects.
- --sitemap-crawl reads the pages in the site's sitemap.xml politely: robots.txt respected, one request a second, its own user agent, and each page fetched once.
- --path='docs/*' imports only the matching paths (shell patterns) and --limit=50 only 50 pages: a quick first look at a large source.
Example 3: a real site, imported in 37 seconds: Learn X in Y minutes
doc.exponential.earth/learninminutes is Learn X in Y minutes (CC BY-SA 3.0): 201 programming language tours, imported straight from GitHub into a siteaccess with its own content tree. This is everything it took.
1. The import type, in settings/override/statichtmlimport.ini.append.php:
[ImportTypes] TypeList[learnx]=expStaticImportTypeLearnX [Type_learnx] SourceRepository=https://github.com/adambard/learnxinyminutes-docs.git ParentRemoteID=learninminutes_site_root Suffixes[] Suffixes[]=md IncludePaths[] IncludePaths[]=*.md ExcludePaths[] ExcludePaths[]=README.md ExcludePaths[]=CONTRIBUTING.md DefaultClass=exp_doc_page AttributeMap[] AttributeMap[exp_doc_page/title]=title AttributeMap[exp_doc_page/body]=body AttributeMap[exp_doc_page/original_html]=original_html AttributeMap[exp_doc_page/source_path]=source_path
- SourceRepository: the import clones the repository itself and pulls it on every later run.
- IncludePaths[]=*.md takes the English pages at the top; the translations live in folders (de/, fr/ ...), and a pattern like de/*.md imports one of them.
- exp_doc_page is a class of exp_doc; any class with a title and a rich text body works (article/title, article/body).
2. The one complication, and its easy fix. These pages have no title in their front matter, and their first "# " line is a code comment, so the dry run named the Python page "Single line comments start with a number symbol." A type of your own, in an extension of your own, names each page by the front matter's name instead. Save it as extension/learninminutes/classes/expstaticimporttypelearnx.php, add ActiveExtensions[]=learninminutes to settings/override/site.ini.append.php, and run php bin/php/ezpgenerateautoloads.php -e:
<?php
class expStaticImportTypeLearnX extends expStaticImportTypeMarkdown
{
public function identifier()
{
return 'learnx';
}
public function parse( $source, $path )
{
$page = parent::parse( $source, $path );
list( $front ) = self::splitFrontMatter( $page['original_html'] );
$meta = self::frontMatter( $front );
$name = isset( $meta['name'] ) ? $meta['name'] : ( isset( $meta['language'] ) ? $meta['language'] : '' );
if ( is_string( $name ) && trim( $name ) !== '' )
$page['title'] = trim( $name );
return $page;
}
}3. A siteaccess with its own content tree root:
- A section page (or folder) for the site's root, below the Websites node, with the remote id learninminutes_site_root.
- settings/siteaccess/learninminutes/: a copy of your front siteaccess with content.ini [NodeSettings] RootNode, site.ini IndexPage and DefaultPage (/content/view/full/<root node>), PathPrefix (the root's URL path), SiteURL and SiteName set for the new site.
- learninminutes added to SiteList[], AvailableSiteAccessList[] and RelatedSiteAccessList[] in settings/override/site.ini.append.php.
- The Anonymous role's user/login policy allowed on the new siteaccess, or visitors are refused.
- Caches cleared: php bin/php/ezcache.php --clear-tag=ini, and with DesignLocationCache=enabled also --clear-id=design_base,template,template-override.
4. Look first, then import:
php bin/php/console exp:static-import:run --learnx --dry-run php bin/php/console exp:static-import:run --learnx
- Dry run: 201 pages, nothing written, 6.1 seconds.
- Import: 201 pages created, 0 failed, 36.7 seconds.
- Again, nothing changed upstream: 201 unchanged, 5.7 seconds.
- Again, after a rendering fix: only the 141 changed pages updated, 60 unchanged, 21.6 seconds. Nothing to clean up by hand.
Use exp_static_html_imports from 2026-11-07 or later: before it, Markdown pages on Exponential 6.0.15 were stored as the word "Array", and wrapped lines lost their spaces. This import found both; the fix is commit e4a240c.
Your own classes and sources
Copy the type's group from extension/exp_static_html_imports/settings/statichtmlimport.ini.append.php into settings/override/statichtmlimport.ini.append.php and change it. For example, HTML pages as Articles:
[Type_html-generic] SourceLocation=/path/to/html ParentNode=2 DefaultClass=article AttributeMap[] AttributeMap[article/title]=title AttributeMap[article/body]=body
- Source: SourceLocation (a folder) or SourceRepository (a git URL the import clones and pulls).
- Place: ParentNode, or ParentRemoteID for a node with that remote id.
- Classes: DefaultClass, ClassRules[] (a class per path pattern) and AttributeMap[] (class/attribute=field).
- Rules are INI by default, or YAML, or a PHP rule class (RulesFormat).
- A source type of your own is a PHP class implementing expStaticImportTypeInterface, registered in statichtmlimport.ini [ImportTypes] TypeList[].
- Every rule is described in the extension's doc/rules.md, also shown in the help center when the extension is active.
Example 4: your own copy of the doc.ez.no archive
git clone https://github.com/se7enxweb/exp_static_html_imports.git extension/exp_static_html_imports git clone https://github.com/se7enxweb/exp_doc.git extension/exp_doc
Activate both in settings/override/site.ini.append.php, in this order:
[ExtensionSettings] ActiveExtensions[]=exp_static_html_imports ActiveExtensions[]=exp_doc
Give the folder that will be the documentation site's root the remote id exp_doc_site_root, then:
php bin/php/ezpgenerateautoloads.php -e php bin/php/ezcache.php --clear-all php bin/php/console exp:migration:run --extension=exp_doc php bin/php/console exp:doc:import --path='FAQ/*' --dry-run php bin/php/console exp:doc:import --path='FAQ/*' php bin/php/console exp:doc:import
- The migrations create the class group Documentation with its classes (pages, sections, manuals, versions, tutorials and their steps, FAQ groups and their questions, extension reference pages, images and files, each with a Bootstrap icon and a metadata attribute), the sections Doc, Media and Hidden, the role Doc reader for anonymous read access, and the media folder.
- exp:doc:import clones the archive from its repository by itself and imports it with the documentation's rules (exp_doc/settings/statichtmlimport.ini.append.php). It takes the same options as exp:static-import:run.
- --path='FAQ/*' imports one part in minutes. The whole archive is 88,680 pages: its final run took 69 minutes on our server. An interrupted run continues where it stopped when you run the same command again.
- Design: set SiteDesign=doc (Modern) or SiteDesign=doc2025 (Classic) in the front siteaccess's site.ini.append.php. The header switches between them with one click.
- Cronjobs: run runcronjobs.php frequent every five minutes. It reads the front page's commit and forum feeds and recounts the Contributors page; without it both still work, more slowly.
- SQLite: after a large import run PRAGMA analysis_limit = 1000; ANALYZE; again (a migration does it once). It changes no data and keeps content fetches fast.
- Duplicate contributors: php bin/php/console exp:doc:contributor-merge --into=<contributor> --from=<duplicate> [--dry-run].
One run is enough; running again is safe
- Identity: every page is the object with remote id staticimport-<type>-<md5 of its source path>, wherever it has been moved.
- Changed source: a new version is published. Unchanged: nothing is written.
- Edited in the CMS: the page is flagged and left alone; --overwrite-flagged writes it anyway and the edit is lost.
- Gone from the source: the node is hidden and reported; --apply-removals moves it to the trash.
- Copies of a page and redirect stubs become redirects to it; error pages are skipped. Every old path, with and without .html, redirects to its new node.
- Large sources run in batches (--batch, 50 by default) and save their position after each; the same command resumes, --restart starts over, --no-pull skips the git pull.
- Every run writes a JSON report (schema exp-static-import/1), a Markdown summary and audit trail events.
- In the administration, Setup > Static site import shows the runs, starts one as a background job, and opens each report with one click.
Questions, ideas and pull requests are welcome on GitHub. Enjoy!
Add comment