Jump to a tool

Text CleanerBuild a reusable Unicode cleanup, extraction, and formatting pipeline for untidy text and lists.Remove Duplicate LinesDeduplicate lines while controlling case, whitespace, order, and blanks.Remove Empty LinesDelete blank rows or collapse them while preserving paragraph structure.Trim WhitespaceRemove unwanted leading or trailing whitespace with indentation control.Remove Extra SpacesNormalize repeated spaces, tabs, and non-breaking spaces safely.Sort LinesSort text alphabetically, naturally, numerically, or by length.Add Prefix and SuffixWrap, number, or punctuate every line with practical presets.Delimiter ConverterConvert lists between lines, commas, tabs, pipes, and custom separators.Compare Two ListsFind unique, shared, and duplicated items across two lists.Find and ReplaceReplace plain text or guarded regular-expression matches.Case ConverterSwitch text among readable, coding, path, and letter-case formats.Word, Character and Line CounterMeasure words, grapheme characters, lines, sentences, time, and UTF-8 bytes.Join LinesCombine rows with a standard or custom separator and optional quoting.Split Text into LinesBreak delimited or regex-matched text into clean one-value-per-line output.Number LinesAdd numbers, letters, Roman numerals, bullets, checklists, or custom markers.Split Text into ChunksCreate overlapping character, word, line, or paragraph chunks with labels.Email ExtractorFind practical email patterns, filter domains, and summarize results.URL ExtractorExtract web URLs, normalize components, filter hosts, and count domains.Number ExtractorExtract localized numbers, percentages, currency-like values, and optional phones.Regex ExtractorExtract whole matches or capture groups with positions and a guarded worker.Invisible Character DetectorFind zero-width, directional, control, and unusual spacing characters with exact positions.Invisible Character RemoverConservatively remove selected hidden characters without breaking emoji joiners by default.Unicode NormalizerNormalize text to NFC, NFD, NFKC, or NFKD and compare changed code-point positions.Unicode Code Point InspectorInspect characters or graphemes with code points, encodings, categories, positions, and export.JSON Formatter and ValidatorValidate JSON, pinpoint syntax errors, and format it with controlled indentation and Unicode output.JSON MinifierValidate JSON and remove insignificant whitespace while preserving string values exactly.JSON to CSVConvert an array of JSON objects to previewable, configurable CSV.CSV to JSONParse quoted CSV safely and convert rows to JSON objects or arrays.CSV Viewer and CleanerPreview, filter, reshape, clean, sort, and export CSV entirely in a Web Worker.Base64 Encode and DecodeEncode or decode UTF-8 text with standard Base64 or Base64URL.URL Encoder and DecoderEncode full URLs or URI components and decode percent-encoded text.HTML Entity Encoder and DecoderEscape HTML special characters or decode named and numeric entities.JWT DecoderDecode JWT header and payload claims without verifying the signature.Unix Timestamp ConverterConvert epoch seconds, milliseconds, and ISO date strings.UUID Generator and InspectorGenerate UUID v4 values in bulk or inspect UUID version and variant.HTML to Plain TextRemove HTML tags while preserving readable text, links, lists, and alt text.HTML to MarkdownConvert common HTML document structure into Markdown source.Markdown to HTMLConvert Markdown into raw HTML and a sanitized preview source.Markdown Table GeneratorTurn CSV, TSV, or pasted spreadsheet rows into Markdown table syntax.HTML Table to CSVExtract pasted HTML table markup into CSV or TSV rows.JSON to YAMLConvert valid JSON into readable YAML with indentation and quoting controls.YAML to JSONConvert common YAML mappings and sequences into JSON.YAML Formatter and ValidatorValidate and format common YAML with indentation and duplicate-key warnings.XML Formatter and ValidatorCheck whether XML is well formed and pretty-print or minify it.XML to JSONMap XML elements, attributes, and text nodes into JSON.Filter LinesKeep or remove lines using text, regex, numeric, or length rules.Column ExtractorExtract selected columns from CSV, TSV, pipe, whitespace, or custom-delimited text.Randomize LinesShuffle lines, take a random sample, or create repeated shuffled copies.Reverse LinesReverse line order, characters, whole text, or word order safely.Word Frequency CounterCount repeated words, unique words, ranks, counts, and percentages.CSV Validator and Data ProfilerInspect CSV structure, inferred types, empty cells, cardinality, and formula-like values.CSV Join and MergeJoin two CSV datasets by one or more keys with explicit duplicate and join rules.CSV Diff ViewerCompare two CSV datasets by key or row position and show changed cells.CSV Pivot and UnpivotReshape CSV data between long and wide layouts with transparent aggregation rules.CSV SplitterSplit a CSV by row count, approximate size, or column value and download one ZIP.CSV TransposeSwap CSV rows and columns while preserving uneven values with explicit labels.CSV Data MaskerRedact, alias, or pseudonymize selected CSV columns locally for safer test data.CSV Encoding ConverterDecode common CSV encodings and produce a UTF-8 file locally.Hash and Checksum Generator/VerifierGenerate or compare SHA-1, SHA-256, SHA-384, and SHA-512 digests locally.HMAC Generator/VerifierGenerate or compare HMAC-SHA-256/384/512 values using a local key.JWT Signature VerifierVerify supported JWT signatures with a local secret, PEM public key, or JWK.XPath TesterEvaluate XPath expressions against pasted XML with optional namespace mappings.OpenAPI Diff ViewerCompare two OpenAPI JSON or YAML documents and report changed specification paths.XML Diff ViewerCompare two XML documents structurally while ignoring attribute order.Semantic Version Range TesterTest semantic versions against common comparator, caret, tilde, and wildcard ranges.JSON5 Formatter and ConverterValidate JSON5, format it, minify it, and convert it to strict JSON locally.JSON Diff ViewerCompare two JSON documents and explain added, removed, changed, and type-changed valuesJSON MergeMerge two or more JSON objects while controlling conflicts, arrays, and null valuesJSON Flatten and UnflattenFlatten nested objects into path-value pairs or reconstruct nested JSON from flat keysJSON Sort KeysSort JSON object keys recursively or at selected depths without changing valuesJSONPath TesterRun JSONPath expressions against pasted JSON and inspect matched values and pathsJSON Pointer EvaluatorResolve, build, and explain RFC 6901 JSON Pointer pathsJSON Schema GeneratorInfer a JSON Schema from one or more sample JSON documentsJSON Schema ValidatorValidate a JSON instance against a pasted JSON Schema and show precise failing pathsNDJSON Formatter and ValidatorValidate one JSON value per line, format records, and locate invalid NDJSON rowsJSON Array and NDJSON ConverterConvert a JSON array to NDJSON or combine NDJSON records into a JSON arrayJSON to TypeScriptInfer TypeScript interfaces or type aliases from sample JSONQuery String Parser and BuilderParse query strings into key-value rows and rebuild encoded query stringsURL Parser and InspectorParse a URL into scheme, credentials, host, port, path, query, and fragmentHTTP Header ParserParse raw HTTP request or response headers and explain common fieldsCookie Parser and FormatterParse Cookie or Set-Cookie headers into names, values, and attributescURL ConverterConvert a pasted cURL command to Fetch or Axios and rebuild supported cURL commandsCron Expression ParserParse cron expressions, explain schedules, and show the next matching timesCron Expression GeneratorGenerate cron expressions from common scheduling controls and preview run timesSemantic Version ComparatorValidate, compare, sort, and explain semantic-version stringsSQL FormatterFormat SQL statements with configurable indentation and keyword caseSQL MinifierRemove unnecessary SQL whitespace and optionally comments without changing quoted stringsCSV to SQL InsertConvert CSV rows into SQL INSERT statements with explicit table and type rulesRegex TesterTest regular expressions, highlight matches, inspect groups, and explain flagsOpenAPI Formatter and ValidatorFormat and validate local OpenAPI JSON or YAML documentsText Diff ViewerCompare two text blocks and highlight inserted, deleted, and changed contentNormalize Line EndingsDetect and convert CRLF, LF, and CR line-ending styles without changing line contentTabs to SpacesConvert tab characters to spaces using a configurable tab widthSpaces to TabsConvert leading groups of spaces to tabs using a chosen indentation widthWrap Text to Column WidthWrap paragraphs to a target column width without breaking words by defaultUnwrap Hard-Wrapped TextRemove hard line breaks inside paragraphs while preserving true paragraph boundariesRemove DiacriticsRemove combining marks and common diacritics while preserving readable base charactersEmoji ExtractorExtract emoji sequences, count them, and preserve their original orderEmoji RemoverRemove emoji sequences while preserving ordinary Unicode text and nearby spacingHashtag ExtractorFind hashtag-like tokens, normalize them, and summarize frequencyMention ExtractorExtract `@username`-style mentions and report unique values and frequencyDomain ExtractorFind domain and hostname values in URLs, email addresses, and plain textDate ExtractorExtract date-like values in selected formats and optionally normalize them to ISO outputPhone Number NormalizerClean punctuation and normalize phone numbers using a selected default countrySlug GeneratorConvert titles or labels into configurable URL-friendly slugsFilename SanitizerReplace or remove characters that are unsafe across common operating systemsHex Encoder and DecoderEncode UTF-8 text as hexadecimal bytes or decode valid hex back to textBinary Text ConverterEncode UTF-8 text as binary bytes or decode binary byte sequences to textBase32 Encoder and DecoderEncode UTF-8 text to Base32 or decode Base32 input with validationBase58 Encoder and DecoderEncode UTF-8 bytes with Base58 and decode valid Base58 stringsUnicode Escape and UnescapeEscape text as Unicode sequences and decode supported escapes back to charactersJavaScript String Escape and UnescapeEscape or unescape text for JavaScript string and template literalsJSON String Escape and UnescapeEscape raw text for a JSON string or decode a valid JSON string valuePunycode and IDN ConverterConvert internationalized domain names between Unicode and Punycode formsROT13 and Caesar CipherEncode or decode ROT13 and configurable Caesar shiftsMorse Code ConverterEncode supported text as Morse code or decode Morse sequences back to textNumber Base ConverterConvert signed integers among binary, octal, decimal, hexadecimal, and custom basesScientific Notation ConverterConvert decimal values to scientific notation and expand exponent notationRoman Numeral ConverterConvert supported positive integers to Roman numerals and validate Roman-numeral inputCSV to HTML TableConvert CSV or TSV into escaped, accessible HTML table markupJSON to HTML TableConvert suitable JSON arrays or objects into escaped HTML tablesTOML Formatter and ValidatorParse TOML, report syntax errors, and generate consistently formatted outputTOML to JSONParse TOML into JSON while documenting datetime and type conversionJSON to TOMLConvert compatible JSON objects to TOML with clear warnings for unsupported shapesINI Formatter and ParserParse common INI syntax, report issues, and produce consistent formattingINI to JSONConvert common INI configuration into JSON using explicit section and value rulesJSON to INIConvert simple JSON objects into INI sections and key-value pairsENV File FormatterFormat dotenv-style files, detect duplicates, and preserve value semanticsENV to JSONConvert dotenv-style key-value entries into JSON without evaluating variable expansionJSON to ENVConvert a compatible JSON object into dotenv-style key-value linesSKU GeneratorGenerate configurable SKU values from product attributes, prefixes, counters, and random segmentsSKU Cleaner and NormalizerNormalize separators, capitalization, whitespace, and allowed characters in SKU listsInvoice Number GeneratorGenerate invoice numbers from prefixes, dates, sequences, and padding rulesUTM Builder and CleanerAdd, inspect, edit, remove, and normalize UTM campaign parametersBulk Filename PlannerCreate a previewable rename plan from pasted filenames without touching local filesFull Name SplitterSplit full-name strings into configurable first, middle, and last-name fieldsContact CSV MergerMerge contact CSV exports using email, phone, or selected identifiersProduct Title CleanerClean product titles by normalizing whitespace, punctuation, case, and repeated promotional phrasesReadability Score CalculatorCalculate common English readability scores and explain their limitationsSyllable CounterEstimate English syllable counts and show uncertain words for manual reviewCloze Deletion GeneratorCreate fill-in-the-blank study text by selecting words or matching patternsFlashcard CSV FormatterFormat question-answer pairs as CSV or TSV for common flashcard applicationsBibTeX Formatter and ValidatorFormat BibTeX entries, detect duplicate keys, and report common field issuesRIS to BibTeXConvert RIS records into BibTeX entries with a visible field-mapping reportBibTeX to RISConvert BibTeX bibliography records into RIS with explicit field mappingDOI and Citation Identifier ExtractorExtract DOI, PMID, arXiv, and other scholarly identifiers from reference textISBN ValidatorValidate ISBN-10 and ISBN-13 check digits and convert compatible valuesReference List DeduplicatorIdentify likely duplicate references using DOI, title, author, and year signalsNumber List StatisticsCalculate count, sum, mean, median, mode, range, variance, and standard deviation from pasted numbersNumber Sequence GeneratorGenerate arithmetic sequences from start, end, step, count, or padding rulesRange Expander and CompressorExpand expressions such as `1-5,8,10-12` or compress integer lists into rangesBitwise CalculatorCalculate AND, OR, XOR, NOT, shifts, and integer representationsIPv4 Number ConverterConvert IPv4 dotted decimal to integer, hexadecimal, and binary forms and backCIDR CalculatorCalculate IPv4 network, broadcast, host range, mask, and address count from CIDR notationDate Range GeneratorCreate a list of dates between two endpoints with configurable intervals and formatsDate List GeneratorGenerate recurring dates from a start date, interval, and occurrence countDuration Parser and FormatterParse human-readable, numeric, and ISO 8601 duration formats and normalize themSMS Character CounterCount characters and estimate GSM-7 or Unicode SMS segmentsEmail Subject Line TesterMeasure email subject and preheader length and preview common truncation rangesWhatsApp Link GeneratorGenerate WhatsApp click-to-chat URLs from a phone number and optional messageCSV Formula Injection SanitizerNeutralize spreadsheet formula-like cells in pasted CSV without uploading the data.CSV Column Mapper and RenamerRename CSV headers and reorder selected columns locally.CSV to Markdown TableConvert a small CSV or TSV paste into a Markdown table.CSV Encoding and BOM InspectorInspect BOM markers, line endings, and byte-oriented clues in pasted CSV text.CSV Data Quality ReportProfile headers, row widths, empty values, and duplicate rows in a CSV paste.TSV and Delimited File ValidatorCheck whether delimited text has consistent columns and balanced quotes.Spreadsheet Date Serial ConverterConvert spreadsheet serial dates to ISO dates using an explicit date system.JSON and CSV Batch PreviewerPreview JSON records or CSV rows as a compact local report.CSP Header Builder and ValidatorBuild and inspect a Content-Security-Policy from pasted directives.SRI Hash GeneratorGenerate a browser-local SHA-256 Subresource Integrity token for pasted asset text.HTTP Security Headers AnalyzerAnalyze pasted HTTP response headers for common security controls.X.509 Certificate DecoderInspect pasted PEM certificate metadata locally.SSH Public Key Fingerprint InspectorInspect SSH public key format and calculate a local fingerprint.JWT Claims and Expiry InspectorDecode JWT claims and explain standard time claims without verifying a signature.JSON Web Key Set InspectorInspect a pasted JWK or JWKS structure without fetching remote keys.Environment Variable Secret ScannerScan pasted .env-style text for likely secrets and risky variable names.Markdown Table of Contents GeneratorGenerate a nested Markdown table of contents from heading lines.Markdown LinterCheck pasted Markdown for common heading, spacing, and list consistency issues.HTML Sanitization PreviewerPreview a conservative sanitized version of pasted HTML.HTML Accessibility CheckerCheck pasted HTML for common image, link, heading, and form labeling issues.Meta Title and Description PreviewerPreview title and description lengths for search and social snippets.robots.txt Generator and ValidatorGenerate and inspect robots.txt directives from pasted path rules.Sitemap XML Generator and ValidatorValidate pasted URLs and generate a simple local sitemap XML document.JSON-LD Formatter and ValidatorFormat pasted JSON-LD and check basic context and type structure.

184 tools available.

Text cleanup

Remove Duplicate Lines

Create a clean unique list without losing the original first occurrence.

Processed locally. Your input and output are never uploaded.

Ready

Input0 lines · 0 characters
Output0 lines · 0 characters
OptionsAdjust how this tool processes your text
Remove Duplicate Lines Options
Press Ctrl/Cmd + Enter to process

Related tools

Text cleanupText Cleaner

Build a reusable Unicode cleanup, extraction, and formatting pipeline for untidy text and lists.

Open tool →List toolsSort Lines

Sort text alphabetically, naturally, numerically, or by length.

Open tool →List toolsCompare Two Lists

Find unique, shared, and duplicated items across two lists.

Open tool →

Tool overview

Remove Duplicate Lines tool overview

This tool compares one line at a time, keeps the first matching line, and reports exactly how many duplicates were removed. Matching rules are independent from output formatting.

Lists that benefit from deduplication

  • Remove repeated SKUs from an export
  • Deduplicate email or keyword lists
  • Clean copied form submissions
  • Prepare a distinct list while retaining source order

Choose how repeated lines match

  1. Paste one item per line.
  2. Choose whether capitalization and surrounding whitespace affect matching.
  3. Keep the original order or sort the final unique list.
  4. Copy or download the deduplicated result.

Case, whitespace, blanks, and output order

  • Case-insensitive matching treats Apple and apple as the same key.
  • Ignore surrounding whitespace matches padded and unpadded versions without rewriting the kept line.
  • Retain empty lines keeps at most one matching blank line.
  • Sort result runs after deduplication.

Keep the first matching line

Before
Alpha
Beta
alpha
Beta
After
Alpha
Beta

Visually similar lines that remain distinct

  • Unicode characters are compared using browser string rules.
  • Visually similar characters may still be different code points.
  • The first matching line is the one preserved.

Browse all text cleanup tools →

Questions

Remove Duplicate Lines FAQ

Can I ignore capitalization?

Yes. Turn off case-sensitive comparison to match uppercase and lowercase variants.

Will the tool reorder my list?

Not unless you enable sorting. Original order is preserved by default.