You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
simon987 0331d46fff
Merge pull request #186 from simon987/dev
2 weeks ago
.github/ISSUE_TEMPLATE Merge pull request #155 from dpieski/patch-4 5 months ago
ci Fix build_arm64.sh (again) 2 weeks ago
docs Update USAGE.md 2 weeks ago
schema web UI rewrite, switch to ndjson.zst index format 2 weeks ago
scripts Basic support for WordPerfect files 2 weeks ago
sist2-vue Add v prefix in version badge 2 weeks ago
src Version bump 2 weeks ago
tests web UI rewrite, switch to ndjson.zst index format 2 weeks ago
third-party Basic support for WordPerfect files 2 weeks ago
.dockerignore Fix .dockerignore, add docker build for arm64 2 weeks ago
.drone.yml Fix docker build for arm64 2 weeks ago
.gitattributes Update .gitattributes 2 weeks ago
.gitignore web UI rewrite, switch to ndjson.zst index format 2 weeks ago
.gitmodules Revert "wip" 1 year ago
CMakeLists.txt web UI rewrite, switch to ndjson.zst index format 2 weeks ago
Dockerfile Install libasan5 in Dockerfile 2 weeks ago
Dockerfile.arm64 Install libasan5 in Dockerfile 2 weeks ago
LICENSE Initial commit (squashed) 2 years ago
README.md Version bump 2 weeks ago

README.md

GitHub CodeFactor Development snapshots

Demo: sist2.simon987.net

sist2

sist2 (Simple incremental search tool)

Warning: sist2 is in early development

sist2.png

Features

  • Fast, low memory usage, multi-threaded
  • Mobile-friendly Web interface
  • Portable (all its features are packaged in a single executable)
  • Extracts text and metadata from common file types *
  • Generates thumbnails *
  • Incremental scanning
  • Manual tagging from the UI and automatic tagging based on file attributes via user scripts
  • Recursive scan inside archive files **
  • OCR support with tesseract ***
  • Stats page & disk utilisation visualization

* See format support
** See Archive files
*** See OCR

stats

Getting Started

  1. Have an Elasticsearch (>= 6.X.X) instance running

    1. Download from official website
    2. (or) Run using docker:
      docker run -d --name es1 --net sist2_net -p 9200:9200 \
          -e "discovery.type=single-node" elasticsearch:7.14.0
      
    3. (or) Run using docker-compose:
        elasticsearch:
          image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
          environment:
            - discovery.type=single-node
            - "ES_JAVA_OPTS=-Xms1G -Xmx2G"
      
  2. Download sist2 executable

    1. Download the latest sist2 release *
    2. (or) Download a development snapshot (Not recommended!)
    3. (or) docker pull simon987/sist2:2.11.2-x64-linux
  3. See Usage guide

* Windows users: sist2 runs under WSL

Example usage

See Usage guide for more details

  1. Scan a directory: sist2 scan ~/Documents -o ./docs_idx
  2. Push index to Elasticsearch: sist2 index ./docs_idx
  3. Start web interface: sist2 web ./docs_idx

Format support

File type Library Content Thumbnail Metadata
pdf,xps,fb2,epub MuPDF text+ocr yes author, title
cbz,cbr (none) - yes -
audio/* ffmpeg - yes ID3 tags
video/* ffmpeg - yes title, comment, artist
image/* ffmpeg - yes Common EXIF tags, GPS tags
raw, rw2, dng, cr2, crw, dcr, k25, kdc, mrw, pef, xf3, arw, sr2, srf, erf LibRaw - yes Common EXIF tags, GPS tags
ttf,ttc,cff,woff,fnt,otf Freetype2 - yes, bmp Name & style
text/plain (none) yes no -
html, xml (none) yes no -
tar, zip, rar, 7z, ar ... Libarchive yes* - no
docx, xlsx, pptx (none) yes if embedded creator, modified_by, title
doc (MS Word 97-2003) antiword yes yes author, title
mobi, azw, azw3 libmobi yes no author, title
wpd (WordPerfect) libwpd yes no planned

* See Archive files

Archive files

sist2 will scan files stored into archive files (zip, tar, 7z...) as if they were directly in the file system. Recursive (archives inside archives) scan is also supported.

Limitations:

  • Support for parsing media files with formats that require seek (e.g. .gif, .mp4 w/ fragmented metadata etc.) is limitted (see --mem-buffer option)
  • Archive files are scanned sequentially, by a single thread. On systems where sist2 is not I/O bound, scans might be faster when larger archives are split into smaller parts.

OCR

You can enable OCR support for pdf,xps,fb2,epub file types with the --ocr <lang> option. Download the language data files with your package manager (apt install tesseract-ocr-eng) or directly from Github.

The simon987/sist2 image comes with common languages (hin, jpn, eng, fra, rus, spa) pre-installed.

Examples

sist2 scan --ocr jpn ~/Books/Manga/
sist2 scan --ocr eng ~/Books/Textbooks/

Build from source

You can compile sist2 by yourself if you don't want to use the pre-compiled binaries

git clone --recursive https://github.com/simon987/sist2/
cd sist2
docker build . -f ./Dockerfile -t my-sist2-image
docker run --rm my-sist2-image cat /root/sist2 > sist2-x64-linux

On a linux computer

  1. Install compile-time dependencies

    apt install gcc g++ python3 yasm ragel automake autotools-dev wget libtool libssl-dev curl zip unzip tar xorg-dev libglu1-mesa-dev libxcursor-dev libxml2-dev libxinerama-dev gettext nasm git
    
  2. Apply vcpkg patches, as per sist2-build Dockerfile

  3. Install vcpkg dependencies

    vcpkg install curl[core,openssl]
    vcpkg install lmdb cjson glib brotli libarchive[core,bzip2,libxml2,lz4,lzma,lzo] pthread tesseract libxml2 libmupdf gtest mongoose libuuid libmagic libraw jasper lcms gumbo
    
  4. Build

    git clone --recursive https://github.com/simon987/sist2/
    cmake -DSIST_DEBUG=off -DCMAKE_TOOLCHAIN_FILE=<VCPKG_ROOT>/scripts/buildsystems/vcpkg.cmake .
    make