Map data pipeline

Everything the server serves — vector tiles, search index, routing graph, terrain, fonts, sprites — is generated into data/ by the Node scripts in data/pipeline/. Nothing under data/raw/, data/build/, data/fonts/, data/routing/ or data/cache/ is committed (.gitignore); the deployment mounts the built directory read-only at /data (.env.prod.example, UZMAPS_DATA).

The primary source is the OpenStreetMap extract for Uzbekistan from Geofabrik. GeoNames, Overture Places, Microsoft building footprints and the Copernicus DEM are supplements. Each has its own licence; see Sources and licences.

What you need#

The pipeline binary is bin/uzmap-new(.exe), not bin/uzmap(.exe): the latter is the binary a running server was started from and Windows refuses to overwrite it. Override with UZMAP_BIN.

Running it#

npm install
npm run data:all                                   # every step, in order
node data/pipeline/all.mjs --list                  # steps, required/optional, one-line description
node data/pipeline/all.mjs --dry-run               # print the plan, run nothing
node data/pipeline/all.mjs --only=overture,index,places
node data/pipeline/all.mjs --from=index --strict   # --strict: exit 2 if an optional step fails
node data/pipeline/furniture.mjs                   # not part of data:all, see below
npm run style:emit                                 # static style JSON, not part of data:all

all.mjs runs each step as a child node data/pipeline/<step>.mjs. A failed required step stops the run and the remaining steps are reported as skipped; a failed optional step is reported and the run continues. Exit code 1 means a required step failed; 2 means only optional steps failed and --strict was given (or an unknown argument was passed). Flags: --dry-run/-n, --only=a,b, --skip=a,b, --from=step, --strict, --list.

Order is not alphabetical and it matters: overture must precede index, which consumes data/raw/overture/places.geojsonl, and places must follow index, which writes its input.

Rerunning#

• Downloads skip a destination that already exists and is non-empty. To refresh the OSM extract or GeoNames, delete the file first. Interrupted downloads leave a .part file that is discarded on the next run; each download retries up to 5 times with 1 s → 30 s backoff, verifies Content-Length when the server sends one, and aborts after 60 s without data.
• overture refuses to overwrite an existing places.geojsonl unless UZMAP_FORCE=1 is set.
• Stop a running uzmap serve before rebuilding any .pmtiles. The server memory-maps the archives, so Windows will not let Planetiler or tippecanoe replace the file, and the tool's own error names neither the lock nor the server. tiles.mjs, buildings.mjs and furniture.mjs check for this and fail with a message that says so (EPERM/EBUSY on the pre-delete).

Steps#

Timings below are from the logs kept in data/*.log and from output-file timestamps in this checkout, on a 16-thread machine running Docker Desktop on Windows. They are indicative, not guarantees.

1. download — OSM extract and GeoNames#

data/pipeline/download.mjs. Fetches three files into data/raw/:

Time: bandwidth-bound; the GeoNames files took about 3 s (data/download.log).

2. fonts — Noto Sans glyphs#

data/pipeline/fonts.mjs. Downloads pre-built SDF glyph ranges for Noto Sans Regular, Noto Sans Medium and Noto Sans Italic from https://raw.githubusercontent.com/protomaps/basemaps-assets/main/fonts/<stack>/<start>-<end>.pbf, 256 ranges per stack (0-255 … 65280-65535), 768 files in total, 24 in parallel. A range the upstream does not have (HTTP 404) is written as an empty file so it is not requested again. Existing files are skipped. Writes data/fonts/LICENSE.txt.

Output: data/fonts/<stack>/<range>.pbf (12 MB). Served at /fonts/. Time: 11 s (data/fonts-download.log).

3. sprites — icon sheets#

data/pipeline/sprites.mjs. Imports badgeIcons, rawIcons and customGlyphs from packages/cartography/src/icons.ts, then renders with sharp:

• badge icons — a circle in the category-group colour with a white glyph, named poi-<name>. The glyph comes from packages/cartography/icons/glyphs/<name>.svg when it is a custom glyph, otherwise node_modules/lucide-static/icons/<name>.svg.
• raw icons — SVGs from packages/cartography/icons/raw/; those flagged sdf are converted to signed-distance fields so icon-color works at runtime. Stretchable icons carry content, stretchX and stretchY in the JSON.

Output: data/sprites/sprite.png, sprite.json, sprite@2x.png, sprite@2x.json, LICENSE.txt. The current sheet has 104 icons, 8 of them SDF. Served at /sprites/. Time: not logged; a local render.

4. overture — Overture Places for the Uzbekistan bbox#

data/pipeline/overture.mjs. Queries the Overture Maps GeoParquet release directly on S3 with DuckDB, run inside python:3.12-slim (pip install duckdb, then a generated data/raw/overture/fetch-places.py). The container is started with --memory=3g --cpus=2; DuckDB is set to memory_limit='2GB', threads=2. The earlier attempt with the overturemaps CLI ran out of memory in the Docker VM, which is why DuckDB is used instead.

Source: s3://overturemaps-us-west-2/release/<release>/theme=places/type=place/* (anonymous, region us-west-2). The bbox struct column is part of the Parquet row-group statistics, so a WHERE bbox.xmin … bbox.ymin … predicate skips every row group outside the bbox and the scan moves a few hundred MB rather than the global theme. The bbox is processed in longitude slices for visible progress and low peak memory.

Output: data/raw/overture/places.geojsonl — one GeoJSON Feature per line, Point geometry, every Overture column kept as a property (id, categories, confidence, websites, emails, socials, phones, brand, addresses, names, sources, operating_status, basic_category, taxonomy, version) — plus places.meta.json (release, bbox, feature count, property list, DuckDB version, fetch time, licence, attribution). pip wheels and DuckDB extensions are cached in data/cache/pip and data/cache/duckdb.

In this checkout: 27,661 places, 37 MB. Time: 94 s inside the container, about 2 min wall (data/overture-download.log).

5. tiles — vector basemap#

data/pipeline/tiles.mjs. Runs Planetiler's OpenMapTiles profile in Docker:

docker run --rm -e JAVA_TOOL_OPTIONS=-Xmx6g -v <data>:/data ghcr.io/onthegomap/planetiler:latest
  --osm-path=/data/raw/uzbekistan-latest.osm.pbf
  --output=/data/build/uzbekistan.pmtiles
  --download --download-dir=/data/raw/planetiler-sources
  --water-polygons-url=https://osmdata.openstreetmap.de/download/simplified-water-polygons-split-3857.zip
  --languages=uz,uz-Cyrl,uz-Latn,ru,en,kaa,kk,tg,de,fr --transliterate=true
  --maxzoom=16 --mmap-temp=false --building-merge-z13=false --force

Heap: UZMAP_PLANETILER_XMX (default 6g). Requires the OSM extract; fails early if it is missing.

Two settings are deliberate:

• --maxzoom=16 rather than Planetiler's default 14. Street zooms overzoom a z14 tile three to four times and magnify the simplification baked in at z14 — which is what put building footprints across roads.
• --mmap-temp=false. At z16 the memory-mapped temp reader overflows its 32-bit offsets once a sort chunk passes 2 GB (NegativeArraySizeException); plain IO has no such limit.

--download fetches the profile's other inputs once into data/raw/planetiler-sources/: natural_earth_vector.sqlite.zip (434 MB, naciscdn.org), lake_centerline.shp.zip (81 MB, github.com/acalcutt/osm-lakelines), water-polygons-split-3857.zip (24 MB, osmdata.openstreetmap.de). Planetiler's scratch directory is data/tmp/, and it leaves data/tile_weights.tsv.gz (5.7 MB) behind.

Output: data/build/uzbekistan.pmtiles, PMTiles v3, z0–16, 543 MB, OpenMapTiles layers aerodrome_label, aeroway, boundary, building, housenumber, landcover, landuse, mountain_peak, park, place, poi, transportation, transportation_name, water, water_name, waterway (read from the archive header and metadata). Served at /tiles/{z}/{x}/{y}.pbf, /tiles/tilejson.json and, for the pmtiles:// protocol, /tiles/uzbekistan.pmtiles with range requests.

Time: the earlier z14 configuration logged 23 min 37 s including first-time source downloads (data/tiles-build.log). The z16 archive in this checkout was written 20 minutes after its OSM extract was downloaded, with sources already cached (file timestamps; no log kept). A comment in furniture.mjs calls the build "hour-long", so allow up to an hour on slower hardware.

Note that docs/architecture.md still describes the tileset as z0–14; the script and the built archive say z0–16.

6. index — search and geocoding#

data/pipeline/index.mjs. Builds the Go server, then runs:

uzmap index --pbf data/raw/uzbekistan-latest.osm.pbf --out data/build/search.sqlite
  [--geonames data/raw/geonames-UZ.zip] [--geonames-alt data/raw/geonames-alternatenames-UZ.zip]
  [--overture data/raw/overture/places.geojsonl --overture-out data/build/places-overture.geojsonl]
  [--datasets datasets --datasets-out data/build/places-custom.geojsonl]

Each optional input is passed only when the file or directory exists. uzmap index also accepts --overture-min-confidence (default 0.5), which the script does not set.

Merge order, and why: OSM entries first; GeoNames enriches OSM places with multilingual names and populations; Overture places are then applied, deduplicating against existing OSM POIs; first-party datasets/ records are merged last — a custom record that duplicates an existing POI (OSM or Overture) replaces it, so a curated record is never shadowed by a bulk one. Overture places are kept only when confidence ≥ 0.5, the category maps onto the UzMap taxonomy, the address country code is empty or UZ, and they do not duplicate an OSM POI.

Logged run (data/index2.log): 1,529 admin/multipolygon relations; 17.2 M nodes cached; 327,431 way entries, 50,041 street ways; 764 admin polygons; 8,306 GeoNames places loaded, 5,800 OSM places enriched; Overture 27,661 read, 11,573 added (7,863 low-confidence, 1,846 unmapped category, 5,396 outside Uzbekistan, 983 duplicates of OSM); 23,223 merged streets; 363,740 entries in 68 s. Earlier runs took 64–81 s. Add the go build.

Outputs:

The server does not read search.sqlite on every start: on first start it builds the in-memory index and writes data/build/search.cache (60 MB here) in the background, and reuses it while the SQLite file's modification time and size are unchanged. Rebuilding the index invalidates the cache automatically.

First-party datasets#

Drop operator-supplied place data in datasets/ (gitignored; "the repo tracks the importer, not the input"). The indexer picks up datasets/*.json and datasets/*/*.json — only .json; the .csv and .geojson siblings in this checkout are ignored. The supplied files are JSON arrays of records. Accepted fields:

Records outside lon 55.9–73.2, lat 37.1–45.6 are dropped as data errors (uzBounds). A record from datasets/dataset_4.json, trimmed:

[
  {
    "id": "place_00031",
    "name": "Nexus",
    "type": "business",
    "longitude": 69.259806,
    "latitude": 41.339461,
    "address": "Tashkent, Takhtapul Darvoza Street, 312",
    "city": "Tashkent",
    "street": "Takhtapul Darvoza Street",
    "house": "312",
    "categories": ["Sale and lease of commercial real estate"],
    "phones": ["+998 94 581 97 97"],
    "schedule": [{ "day": "Monday", "hours": "08:00–17:00" }],
    "gallery": ["images/50629e91a3e982886f13f0c436a676c5.jpg"]
  }
]

To see what an unfamiliar file contains before importing it, node data/pipeline/inspect.mjs <dir-or-file> [--rows=5] reports format, record count, field names with inferred types and fill rates, geometry type and sample rows. It reads at most 8 MB of any file and writes nothing.

7. places — POI overlay tiles#

data/pipeline/places.mjs. Requires data/build/places-overture.geojsonl (fails with a message pointing at data:overture then data:index if absent). Concatenates it with places-custom.geojsonl, when present, into data/build/places-all.geojsonl: one layer keeps the rank-based zoom gating coherent, whereas two overlapping POI layers would each thin themselves independently and look twice as crowded.

The script streams the input once to collect rank values and derives per-zoom thresholds from the rank distribution, then writes a $zoom-aware filter to data/build/places-overture.filter.json. At z14 everything passes; at z13 only the top UZMAP_POI_KEEP_Z13 fraction (default 0.25); at z12 the top UZMAP_POI_KEEP_Z12 (default 0.08); features without a rank pass at every zoom. Set UZMAP_POI_RANK_DESC=1 if a higher rank means more prominent.

tippecanoe is then run with -o /data/build/places-overture.pmtiles -l poi -Z12 -z14 -r1 --no-feature-limit --no-tile-size-limit -J <filter> --order-by=rank --buffer=16 --force. -r1 disables tippecanoe's pseudo-random point thinning so the filter alone decides what survives at low zoom; --order-by=rank puts prominent POIs first in each tile so they win label collisions.

Output: data/build/places-overture.pmtiles (1.4 MB, layer poi, z12–14) and places-overture.meta.json (layer, minzoom, maxzoom, features, rankThresholds, sources, licence, attribution). In this checkout: 14,401 features, thresholds rank <= 3 at both z13 and z12. Time: about 30 s (file timestamps).

8. routing — Valhalla graph#

data/pipeline/routing.mjs. Writes data/routing/build.sh, creates the Docker volume, and runs the script in the Valhalla image:

docker volume create uzmap-valhalla
docker run --rm -v uzmap-valhalla:/valhalla -v <data>:/data --entrypoint bash ghcr.io/valhalla/valhalla:latest /data/routing/build.sh

The graph is built inside a named volume rather than the bind-mounted data/ because Valhalla memory-maps its tiles and SQLite databases, which is unreliable on Docker Desktop bind mounts. The finished valhalla.json and valhalla_tiles.tar are copied to data/routing/ as portable artefacts (239 MB); the running Valhalla container reads the volume, not these copies.

build.sh runs valhalla_build_config (tile dir and extract under /valhalla, timezone and admin SQLite files, concurrency from UZMAP_VALHALLA_THREADS, service limits: auto 2,000,000 m and 50 locations, pedestrian 250,000 m, bicycle 500,000 m, minimum reachability 50), valhalla_build_timezones (which downloads the timezone polygon shapefile) if not already present, valhalla_build_admins, valhalla_build_tiles, valhalla_build_extract.

Time: 1 min 33 s, 901 tiles (data/routing-build.log).

9. buildings — Microsoft ML footprints#

data/pipeline/buildings.mjs. Three stages:

1. Download. Fetches https://minedbuildings.z5.web.core.windows.net/global-buildings/dataset-links.csv, keeps rows whose Location is Uzbekistan (125 parts), downloads each gzipped part and decompresses it to data/raw/ms-buildings/<quadkey>.geojsonl. UZMAP_MS_CONCURRENCY parallel downloads (default 4). Complete parts are skipped on rerun. About 2.4 GB decompressed on disk here.
2. De-duplicate against OSM. uzmap buildings --pbf data/raw/uzbekistan-latest.osm.pbf --ml-dir data/raw/ms-buildings --out data/build/buildings-ml.geojsonl drops footprints that overlap an OSM building, sit on an OSM road corridor, duplicate another ML footprint, or are smaller than --min-area (default 12 m²) (services/server/internal/mlbuildings). Two overlapping extrusions z-fight in the 3D view, which is worse than a missing building. Logged: 1,629,786 OSM footprints as the reference set; 6,504,180 ML footprints read, 5,721,350 kept (670,231 OSM overlaps, 26 ML duplicates, 112,573 tiny) in 291 s (data/ml-merge.log; that run predates the road-corridor test). Output 1.02 GB.
3. Tiles. tippecanoe with -o /data/build/buildings-ml.pmtiles -l building -Z13 -z16 --no-feature-limit --maximum-tile-bytes=6000000 --detect-shared-borders --simplification=4 --buffer=8 --no-simplification-of-shared-nodes -pS --force. z16 rather than z14 for the same overzoom reason as the basemap; the 6 MB tile cap exists because without any cap the z13–14 overview tiles grow unbounded in memory and the run dies against the Docker VM.

Output: data/build/buildings-ml.pmtiles, layer building, z13–16. Time: download about 3 min, merge 4 min 51 s; the z16 tiling is not logged and the archive is absent from this checkout at the time of writing (the server log of 3 September shows an earlier z13–14 build). README.md budgets about 25 min for the whole step.

The tippecanoe image#

places, buildings and furniture share tippecanoe() in _util.mjs. The image is resolved in this order: UZMAP_TIPPECANOE_IMAGE; an already-present ghcr.io/felt/tippecanoe:latest, uzmap/tippecanoe:local or tippecanoe:latest; docker pull of the upstream image (which currently answers "denied" for anonymous pulls, data/tippecanoe-pull.log); otherwise a build from the upstream source tarball into data/raw/tippecanoe-src/ with a generated ubuntu:24.04 Dockerfile, tagged uzmap/tippecanoe:local, taking about 5 min. tippecanoe's temporaries go in the named volume uzmap-tiletmp because they are memory-mapped SQLite-backed files that fail on a Windows bind mount once a build spills to disk.

10. terrain — Copernicus DEM to Terrain-RGB#

data/pipeline/terrain.mjs. Downloads the Copernicus DEM 1°×1° COG cells covering lat 37–45, lon 55–73 (171 cells) from https://<bucket>.s3.amazonaws.com/<name>/<name>.tif, where <name> is Copernicus_DSM_COG_<code>_N<lat>_00_E<lon>_00_DEM. A 404 means a cell with no land data and is not an error.

Cells land in data/raw/dem-90m/ (or dem-30m/). The script then writes data/build/terrain-build.sh and runs it in ghcr.io/osgeo/gdal:ubuntu-small-latest: gdalbuildvrt → gdalwarp to EPSG:3857 over extent 6120000 4430000 8130000 5780000 at 76 m (38 m for GLO-30), bilinear, no-data filled with 0 → three gdal_calc.py passes encoding Mapbox Terrain-RGB (v = (h + 10000) * 10; R = v >> 16, G = (v >> 8) & 255, B = v & 255) → gdal2tiles.py --xyz -z 4-<MAXZ>.

Output: data/build/terrain/{z}/{x}/{y}.png and terrain/meta.json (encoding: "mapbox", minzoom: 4, maxzoom, source, attribution). Served at /terrain/{z}/{x}/{y}.png; /terrain/tilejson.json is built from meta.json and returns 404 with a hint if it is missing. Time: 10 min 44 s for GLO-90 with 53 of 171 cells still to download, of which gdal2tiles took 5 min 28 s (data/terrain-build.log).

furniture — crossings, signals, trees, bus stops (not in data:all)#

data/pipeline/furniture.mjs. There is no data:furniture npm script and all.mjs does not list the step, yet .env.prod.example expects build/furniture.pmtiles and the engine turns the layer on when the server advertises it. Run it by hand:

node data/pipeline/furniture.mjs

It builds the server, runs uzmap furniture --pbf data/raw/uzbekistan-latest.osm.pbf --out data/build/furniture.geojsonl, then tippecanoe with -o /data/build/furniture.pmtiles -l furniture -Z 13 -z 16 --no-feature-limit --maximum-tile-bytes=2000000 --drop-densest-as-needed --force. OpenMapTiles carries none of these point features, and extracting them separately avoids rerunning the basemap build to add three point layers.

Output: data/build/furniture.pmtiles (1.3 MB, layer furniture, z13–16). Time: about 2 min (file timestamps).

emit-style — static style JSON (npm run style:emit)#

data/pipeline/emit-style.mjs calls buildStyle({ theme, language, serverUrl }) from packages/cartography/src/index.ts for light/dark × uz/ru/en and writes data/build/styles/uzmap-<theme>-<language>.json for clients that cannot run the TypeScript style builder (native MapLibre SDKs). UZMAP_SERVER_URL (default http://localhost:8080) is baked into the source URLs. Served at /styles/.

Environment variables#

Output layout#

The engine asks /api/status on load and, for each of sources.buildings_ml, sources.places and sources.furniture that is true, adds the matching /tiles/<name>/tilejson.json source; sources: { mlBuildings, places, furniture } on UzMap can force a URL or disable one. So an optional archive that is present is used, and one that is absent is silently not.

Sources and licences#

Licences below are as recorded in this repository, checked against the upstream licence page where noted. Three points need action and are called out after the table.

Tool licences, from README.md: Planetiler Apache-2.0, Valhalla MIT, MapLibre GL BSD.

Attribution as the system emits it#

Points that need attention#

1. Microsoft footprints are labelled ODbL in the code but licensed CDLA Permissive 2.0 upstream. data/pipeline/buildings.mjs (header and --attribution=© Microsoft Building Footprints (ODbL)) and services/server/internal/mlbuildings/mlbuildings.go say ODbL; the upstream README says CDLA Permissive 2.0 and does not contain the words "ODbL" or "Open Database License"; docs/data-sources.md also says CDLA Permissive 2.0. The licence label embedded in the tile archive is therefore wrong. The visible credit in the style ("© Microsoft Building Footprints", linked to the repository) does not name a licence.
2. Overture Places is not one licence. Overture's attribution page gives no single Places licence: Meta, Microsoft and PinMeTo contributions are CDLA Permissive 2.0, Foursquare's are Apache 2.0 (with a NOTICE.txt Overture links to), AllThePlaces is CC0 1.0. The raw extract in this checkout carries sources[].dataset values: meta 26,768, AllThePlaces 443, Foursquare 342, Microsoft 106, PinMeTo 2 (every record also lists Overture). overture.go does not read sources, and places-overture.geojsonl has no such property, so once indexed there is no way to tell which of the 11,573 kept places came from which provider. Overture's page recommends the citation "Overture Maps Foundation, overturemaps.org".
3. The style credits OpenStreetMap but not OpenMapTiles. The Planetiler output and the archive metadata require "© OpenMapTiles © OpenStreetMap contributors"; style.ts declares only OpenStreetMap on the basemap source, while the TileJSON the same source points at carries both. Which one MapLibre displays when a source has both an inline attribution and a TileJSON attribution has not been checked here.

The Copernicus credit in terrain/meta.json is an abbreviation of the full wording quoted in terrain.mjs and docs/data-sources.md.