Misbar analyzes, cleans, and validates large-scale CSV datasets in the browser — purpose-built for Saudi national identifiers, mobile numbers, and Arabic text alongside standard international formats.
| national_idSaudi National ID | mobileSaudi Mobile | cityText · Arabic | joinedTimestamp | status |
|---|
A secure Flask backend paired with a fully responsive, framework-free JavaScript frontend delivers a desktop-grade data-preparation experience directly in the browser.
Flask application handling routing, session security, permission enforcement, and the CSV upload endpoint.
Delimiter sniffing, 19-class type inference, and index-mapped row formatting for compact network payloads.
Whitespace, casing, fill/blank, fuzzy clustering, sandboxed expression transforms, and Unicode/Arabic/date/number/currency normalization.
Split, join, derive, rename, remove, and move column operations — with facets and rules cleaned up automatically.
Numeric bins, box-plot statistics, scatterplot coordinates, duplicate detection, and text filtering — computed server-side.
Parameterized queries with automatic MySQL-to-SQLite fallback and schema self-upgrade on startup.
A 5,500+ line asynchronous single-page state machine covering the grid, facets, validation, statistics, DQ scoring, history, and administration UI.
Uploads are sniffed for delimiter, stripped of blank rows, and converted to a compact index-mapped structure. Then every column's first 100 non-empty values run through a cascade of pattern detectors.
A paginated grid — 100 rows per page — with sticky headers, drag-resizable columns, and per-cell status highlighting, surrounded by tool panels you can rearrange to fit your screen.
Every header exposes a single dropdown with every relevant facet, cleaning, schema, and validation action for that column's detected type.
Select several columns and a floating action bar appears — validate or facet them together in one step.
Live status pills — All good, Worth a look, Needs fixing, or Pending — reflecting each row's worst active validation outcome.
Collapsible, drag-resizable panels (220–700px) for Facets, Rules, History, Export, Stats, and DQ Dimensions — with an off-canvas drawer below 860px.
Both Bars, Left Only, or Right Only layouts, plus light and dark themes persisted via localStorage.
A bottom-center toast confirms success, surfaces errors, and explains permission-denied actions throughout the app.
Facets are computed server-side for performance and rendered as interactive panels — each with its own PDF export.
Unique values with counts and frequencies, sortable by name or count — click a term to filter the grid.
Groups values into 12 equal-width histogram bins to reveal distribution and skew.
Tukey five-number summary with 1.5× IQR whiskers, outlier detection, and a one-click "Filter outliers" control.
Plots paired values between two numeric columns, aggregating repeated coordinates into density counts.
Surfaces every value occurring more than once, sorted by descending frequency.
Substring or regular-expression matching with case-sensitivity and invert-match toggles.
Two tiers of column-wide operations — applied instantly, automatically re-validated, and recorded in the undo/redo history.
Split values into adjacent columns via a string or regex delimiter, with an optional limit and source-column removal.
Combine multiple selected columns with a chosen delimiter into one new, named column.
Compute a new column from a Python expression evaluated against other cells in each row.
Relabel, delete, or reposition columns — associated facets and validation rules are cleaned up automatically.
Reclassify a column's detected type from the context menu, instantly refreshing its default rules and reports.
Attach rules to one or many columns at once. Export rule sets as a CSV template and re-import them in bulk to replicate a data-quality policy across similar datasets.
| Rule | Class | Checks |
|---|---|---|
| Not empty | Core | Flags blank / null / whitespace-only cells |
| Pattern (regex) | Core | Match against a custom regular expression |
| Number range | Core | Numeric value within a min–max range |
| Allowed values | Core | Value is in a semicolon-separated whitelist |
| Unique (no duplicates) | Core | No repeated values in the column |
| Expression (GREL) | Core | Custom boolean expression per row |
| Valid Email / URL / Phone | Format | Standard format syntax checks |
| Valid Saudi National ID / Iqama / Mobile | Regional (KSA) | Structural checks for Saudi identifiers and numbers |
| Valid Latitude / Longitude | Geospatial | Numeric within valid geographic bounds |
| Valid Date / Timestamp | Temporal | Parses as a real calendar date or date-time |
| Valid Currency / Boolean / JSON / UUID | Format | Structural / semantic format checks |
| Valid IP / MAC / Postal Code | Network & Format | Structural pattern checks |
Cell backgrounds turn red, orange, or blue with hover tooltips explaining exactly why a value failed.
Aggregates issue counts per column and per rule, expandable down to the specific failing row indices.
Export a rule set as CSV and re-import it in bulk — one quality policy, many datasets.
A defensible, quantified quality score — the equal-weighted average of ten dimensions, rendered as color-coded progress bars and exportable as CSV or a formatted PDF scorecard.
A separate Statistical Functions panel rolls a subset into a weighted Quality Score — Completeness 30% + Consistency 30% + Accuracy 40% — for a fast per-column health check.
Session-based authentication with PBKDF2-SHA256 password hashing and HTTP-only, SameSite cookies. Every sensitive server route is wrapped in a permission decorator, and superadmins manage access from an in-app console — no direct database access required.
| Permission | Grants access to |
|---|---|
| can_upload | Uploading new CSV files |
| can_facet | Building and viewing facets / filters |
| can_edit_cell | Standard cell cleaning operations |
| can_edit_column | Split, join, rename, remove, move, derive columns |
| can_advanced_clean | Unicode / Arabic / encoding / date / number / currency normalization |
| can_validate | Creating and running validation rules |
| can_view_stats | Viewing the Descriptive Statistics panel |
| can_view_dq | Viewing the Data Quality Dimensions panel |
| can_export_report | Exporting reports and datasets |
| is_superadmin | Bypasses all checks; exclusive User Management access |
CSV, TSV, or JSON — scoped to the full file, valid rows only, errors only, or warnings only.
A shareable CSV or print-ready PDF of validation results for audit or handoff.
Both analytical panels export independently to CSV or a styled PDF scorecard.
Print the active histogram, box plot, or scatterplot to a PDF-ready layout.
Every mutating action captures a full dataset snapshot. Click any History entry to jump directly to that exact point — not a linear undo stack.
Aggressive transforms are safe when any state is one click away. That confidence is the point of the timeline.
| Layer | Technology | Role |
|---|---|---|
| Backend | Flask 3.0 (Python) | Routing, sessions, REST-style JSON APIs |
| Database (primary) | MySQL via PyMySQL | Production multi-user persistence |
| Database (fallback) | SQLite 3 | Automatic offline / dev fallback |
| Security | Werkzeug Security · python-dotenv · cryptography | Password hashing, secrets, session cookies |
| Frontend | Vanilla JavaScript (ES6+) | Single-page state machine, no framework |
| Styling | CSS custom properties · Plus Jakarta Sans | Light/dark theming, responsive layout |
| Templating | Jinja2 (Flask render_template) | Server-rendered shell pages |
Identified during the 2.0 review and tracked on the product roadmap — because a data-accuracy platform should be accurate about itself.
Misbar 2.0 is internal product documentation made real — upload a CSV, watch 19 detectors classify it, and walk away with a defensible quality scorecard.
Request access