Version: 1.0.0 Last Updated: 2026-05-14 Status: Active

bast-process-pdf¶

Version: 0.2.2 Module: bast_process_pdf Path: src/pipelines/process_pdf

Overview¶

Unified PDF processing pipeline combining basic PDF conversion, advanced text extraction with multiple chunking strategies, and OWL ontology generation capabilities. The most feature-rich pipeline in the platform.

Features¶

Basic PDF Processing¶

PDF to text conversion
PDF to image conversion (PNG from pages)
Page extraction as separate PDFs
JSON output with page data

Advanced Text Extraction¶

Multiple chunking strategies
Document type detection (academic, handbook, report)
Semantic boundary analysis
Cross-page chunking with metadata
Table of Contents detection and filtering
Max chunk size limits (1000 characters)

OWL Ontology Generation¶

AI-powered entity extraction using OpenAI
Structured knowledge representation
Metadata integration (pages, chunks, sources)
Entity deduplication
Hierarchical ontologies
Multiple relationship types
Parallel processing support
Progress tracking callbacks

Pipeline Actions¶

Action	Description
`convert_pdf`	Basic PDF conversion to text/images/JSON
`extract_text`	Advanced text extraction with chunking
`generate_owl`	Generate OWL ontology from processed content
`full_pipeline`	Complete pipeline (extract + generate OWL)

Main Services¶

Service	Description
`ConvertPDFFiles`	Basic PDF conversion
`PDFProcessor`	Advanced text extraction
`OWLGenerator`	OWL ontology generation
`ChunkingStrategyFactory`	Chunking strategy creation

Chunking Strategies¶

Strategy	Description	Best For
`paragraph`	Split by paragraph boundaries	General documents
`content_type`	Aware of headers, lists, code	Technical docs
`semantic`	AI-powered semantic splits	Complex narratives
`fixed_size`	Fixed character count	Uniform processing

Dependencies¶

Internal Dependencies¶

Package	Purpose
bast-ai-base	Core utilities
bast-ai-config	Configuration management
bast-ai-oracle	Database connectivity

External Dependencies¶

Package	Purpose
PyPDF2	PDF processing
PyMuPDF	Advanced PDF operations
pdf2image	PDF to image conversion
pillow	Image manipulation
pycryptodome	Encryption support

Usage Examples¶

Basic PDF Conversion¶

from bast_process_pdf.svc import ConvertPDFFiles

converter = ConvertPDFFiles()
result = converter.convert(
    input_path="document.pdf",
    output_dir="output/",
    extract_text=True,
    extract_images=True,
    generate_json=True
)

Advanced Text Extraction¶

from bast_process_pdf.svc import PDFProcessor

processor = PDFProcessor(catname="mycat")
chunks = processor.extract_text(
    input_path="document.pdf",
    chunking_strategy="semantic",
    max_chunk_size=1000
)

for chunk in chunks:
    print(f"Page {chunk['page']}: {chunk['text'][:100]}...")

OWL Generation¶

from bast_process_pdf.svc import OWLGenerator

generator = OWLGenerator(catname="mycat")

def progress_callback(current, total, message):
    print(f"Progress: {current}/{total} - {message}")

result = generator.generate(
    input_path="document.pdf",
    output_path="output/ontology.owl",
    progress_callback=progress_callback,
    parallel=True
)

Full Pipeline¶

from bast_process_pdf.bp import ProcessPDFPipeline

pipeline = ProcessPDFPipeline(catname="mycat")
result = pipeline.process(
    action="full_pipeline",
    input_path="document.pdf",
    output_dir="output/",
    chunking_strategy="semantic",
    generate_owl=True
)

Entity Relationship Types¶

Relationship	Description
`subClassOf`	Type/subcategory hierarchy
`equivalentTo`	Entity matching for deduplication
`instanceOf`	Specific instance of a class
`relatedTo`	General association
`disjointWith`	Mutually exclusive entities
`complementOf`	Complement relationships

Testing¶

Unit Tests (4 files)¶

test_pipeline_api.py
test_pipeline1.py
test_pdf_processor.py
test_owl_generator.py

Integration Tests (10 files)¶

pdf_conversion.py - Basic conversion tests
pdf_chunking.py - Chunking strategy tests
chunking_factory_usage.py - Factory pattern demo
owl_generation.py - OWL generation tests
json_to_owl.py - JSON to OWL conversion
owl_generator_comprehensive_test.py - Comprehensive tests
owl_generator_io_consistency.py - I/O consistency
test_runner_owl_generator.py - OWL generator runner
test_runner_pymupdf.py - PyMuPDF processor runner

Running Tests¶

# Unit tests
pants test src/pipelines/process_pdf:: -- -s --no-header

# Integration tests
pants run src/pipelines/process_pdf/integration/pdf_conversion.py
pants run src/pipelines/process_pdf/integration/owl_generation.py

Issues & Improvements¶

Current Issues¶

Memory usage: Large PDFs can consume significant memory
Processing speed: Sequential page processing can be slow
OCR support: Limited OCR for scanned documents

Suggested Improvements¶

Streaming processing: Process pages in chunks to reduce memory
GPU acceleration: Use GPU for faster image processing
OCR integration: Add Tesseract or cloud OCR support
Parallel page processing: Process pages concurrently
Better table extraction: Improve table detection and extraction
Resume capability: Save progress for large documents

Configuration¶

# resources/config/{catname}.yaml
process_pdf:
  chunking:
    default_strategy: semantic
    max_chunk_size: 1000
  owl:
    model: gpt-4o
    parallel: true
    batch_size: 10
  output:
    url_prefix: "https://cdn.example.com/pages/"

Changelog¶

v0.2.2 (Current)¶

Added JSON file generation in convert_pdf action
Fixed Unicode encoding issues in JSON output
Added URL prefix configuration for page image references
Enhanced reference generation format

v0.2.1¶

Added OWL generator parallel processing
Improved chunking strategies

v0.2.0¶

Major refactor with new service architecture
Added ChunkingStrategyFactory
Improved entity deduplication

v0.1.0¶

Initial release with basic PDF conversion