diff --git a/README b/README index 34ce202..81374da 100755 --- a/README +++ b/README @@ -15,7 +15,7 @@ gem install docsplit For documentation, usage, and examples, see: - http://documentcloud.github.com/docsplit/ + https://documentcloud.github.io/docsplit/ To suggest a feature or report a bug: http://github.com/documentcloud/docsplit/issues/ diff --git a/docsplit.gemspec b/docsplit.gemspec index a5ab36f..0a147e9 100755 --- a/docsplit.gemspec +++ b/docsplit.gemspec @@ -1,7 +1,7 @@ Gem::Specification.new do |s| s.name = 'docsplit' s.version = '0.7.6' # Keep version in sync with docsplit.rb - s.date = '2014-11-14' + s.date = '2014-11-17' s.homepage = "http://documentcloud.github.com/docsplit/" s.summary = "Break Apart Documents into Images, Text, Pages and PDFs" diff --git a/index.html b/index.html index 982a8f5..ccbcb95 100755 --- a/index.html +++ b/index.html @@ -87,7 +87,7 @@
-

Doc⚡split

+

Docsplit

Docsplit @@ -149,7 +149,7 @@

Installation & Dependencies

(Optional) Install pdftk. On Linux, use aptitude, apt-get or yum:
aptitude install pdftk
- On the Mac, you can download a recent installer for the binary. + On the Mac, you can download a recent installer for the binary. Without pdftk installed, you can use Docsplit, but won't be able to split apart a multi-page PDF into single-page PDFs. @@ -159,6 +159,12 @@

Installation & Dependencies

aptitude install libreoffice
On the Mac, download and install the latest release. +
  • + (Optional) Install fonts to process documents that use Chinese, Japanese, and Korean Fonts. + On Linux, use aptitude, apt-get or yum:
    + aptitude install ttf-wqy-microhei ttf-wqy-zenhei ttf-kochi-gothic ttf-kochi-mincho fonts-nanum
    + On the Mac, the fonts should already be present. However you can always download the TTF files and install them using Font Book. +

    • @@ -183,7 +189,7 @@

    Usage

    and format. Pass --pages or -p to choose the specific pages to image. Passing
    --size or -s will specify the desired image resolution, --density or -d will specify the DPI to rasterize the images - at during conversion by GraphicsMagick, and --format or -f + at during conversion by GraphicsMagick, and --format or -f will select the format of the final images. 

    @@ -201,7 +207,7 @@ 
    Usage
    
       pass --pages all. You can use the --ocr and --no-ocr
       flags to force OCR, or disable it, respectively. By default (if Tesseract is installed)
       Docsplit will OCR the text of each page for which it fails to extract text
-      directly from the document. Docsplit will also attempt to clean up garbage 
+      directly from the document. Docsplit will also attempt to clean up garbage
       characters in the OCR'd text — to disable this, pass the
       --no-clean flag.
     
    
@@ -272,7 +278,7 @@ 
    Internals
    
       Poppler,
       PDFTK,
       Tesseract, and
-      LibreOffice libraries. 
+      LibreOffice libraries.
       Poppler is used to extract text and metadata from PDF documents,
       PDFTK is used to split them apart into pages, and GraphicsMagick is used to generate
       the page images (internally, it's rendering them with
@@ -291,7 +297,7 @@ 
    Internals
    
     
    
 
     
    Change Log
    
-    
+
     
    
       0.7.6 – Nov. 16, 2014
    
       Docsplit will now automatically use Tesseract's orientation detection model
@@ -308,7 +314,7 @@ 
    Change Log
    
       0.7.2 – Feb. 23, 2013
    
       Bug fixes for LibreOffice support.
     
    
-    
+
     
    
       0.7.0 – Feb. 23, 2013
    
       Docsplit now expresses a preference for LibreOffice over OpenOffice, with
@@ -317,81 +323,81 @@ 
    Change Log
    
       Improved unicode support now correctly collects non-ascii characters from
       pdfinfo.
     
    
-    
+
     
    
       0.6.4 – Nov. 12, 2012
    
       Added a language flag for the Docsplit commandline, fixed several bugs,
       and began preparations for the deprecation of pdftk.
     
    
-    
+
     
    
       0.6.2 – Nov. 22, 2011
    
       Bugfix to escape document names during file type detection.
     
    
-    
+
     
    
       0.6.1 – Nov. 18, 2011
    
       Docsplit now supports converting documents using LibreOffice
       as well as OpenOffice, through JODConverter 3.0 beta4.
     
    
-    
+
     
    
       0.6.0 – Sept. 13, 2011
    
-      Docsplit should now handle shelling out for documents with arbitrary 
-      characters in their filenames correctly, thanks to a series of 
+      Docsplit should now handle shelling out for documents with arbitrary
+      characters in their filenames correctly, thanks to a series of
       epic patches from Vladimir Rybas.
-      A --density option was added for specifying the resolution of 
+      A --density option was added for specifying the resolution of
       rasterization when generating images from documents.
       The image resolution for OCR has been doubled from 200 to 400 DPI —
-      this shouldn't make a noticeable difference for normal docs, but will make 
+      this shouldn't make a noticeable difference for normal docs, but will make
       a world of difference for the fine print.
       Docsplit now uses GraphicsMagick's --despeckle before OCR.
     
    
-    
+
     
    
       0.5.2 – May 13, 2011
    
       For transparent conversion to PDF, made Docsplit prefer GraphicsMagick
       over OpenOffice, when the file format is one that GraphicsMagick is able
       to read: (png, gif, jpg, jpeg, tif, tiff, bmp, pnm, ppm, svg, eps).
     
    
-    
+
     
    
       0.5.1 – April 26, 2011
    
       Minor tweaks to the TextCleaner to be more lenient about acryonms
       with hyphens, and words with four vowels in a row.
     
    
-    
+
     
    
       0.5.0
    
       Added a Docsplit::TextCleaner class which is used to post-process
       OCR'd text, and remove garbage characters that are created when Tesseract
       encounters non-english text. To disable the cleanup, pass --no-clean.
     
    
-    
+
     
    
       0.4.1
    
       Upgraded the JODConverter dependency for PDF conversion via OpenOffice to
-      3.0 beta. Added PNG, GIF, TIF, JPG, and BMP to the list of supported 
+      3.0 beta. Added PNG, GIF, TIF, JPG, and BMP to the list of supported
       formats.
     
    
-    
+
     
    
       0.3.4
    
       Adding a suggested optimization from the GraphicsMagick list -- only ever
       generate one page image per GraphicsMagick call. Saves large amounts of
       disk space for tempfiles on long documents.
     
    
-    
+
     
    
       0.3.3
    
       Start using the MAGICK_TMPDIR environment variable to prevent parallel
       Docsplit runs from having the potential to clobber each other's temporary
       image files.
     
    
-    
+
     
    
       0.3.1
    
-      Added a memory limit to GraphicsMagick while generating the TIFFs for 
+      Added a memory limit to GraphicsMagick while generating the TIFFs for
       Tesseract OCR -- prevents gm from gobbling up all available memory
       on large files.
     
    
diff --git a/lib/docsplit/image_extractor.rb b/lib/docsplit/image_extractor.rb
index 8c29bbc..8bc4d1d 100755
--- a/lib/docsplit/image_extractor.rb
+++ b/lib/docsplit/image_extractor.rb
@@ -33,7 +33,7 @@ def convert(pdf, size, format, previous=nil)
       directory = directory_for(size)
       pages     = @pages || '1-' + Docsplit.extract_length(pdf).to_s
       escaped_pdf = ESCAPE[pdf]
-      FileUtils.mkdir_p(directory) unless File.exists?(directory)
+      FileUtils.mkdir_p(directory) unless File.exist?(directory)
       common    = "#{MEMORY_ARGS} -density #{@density} #{resize_arg(size)} #{quality_arg(format)}"
       if previous
         FileUtils.cp(Dir[directory_for(previous) + '/*'], directory)
@@ -48,7 +48,7 @@ def convert(pdf, size, format, previous=nil)
         end
       end
     ensure
-      FileUtils.remove_entry_secure tempdir if File.exists?(tempdir)
+      FileUtils.remove_entry_secure tempdir if File.exist?(tempdir)
     end
 
 
diff --git a/lib/docsplit/page_extractor.rb b/lib/docsplit/page_extractor.rb
index 145c980..0aef939 100644
--- a/lib/docsplit/page_extractor.rb
+++ b/lib/docsplit/page_extractor.rb
@@ -10,15 +10,15 @@ def extract(pdfs, opts)
       [pdfs].flatten.each do |pdf|
         pdf_name = File.basename(pdf, File.extname(pdf))
         page_path = ESCAPE[File.join(@output, "#{pdf_name}")] + "_%d.pdf"
-        FileUtils.mkdir_p @output unless File.exists?(@output)
-        
+        FileUtils.mkdir_p @output unless File.exist?(@output)
+
         cmd = if DEPENDENCIES[:pdftailor] # prefer pdftailor, but keep pdftk for backwards compatability
           "pdftailor unstitch --output #{page_path} #{ESCAPE[pdf]} 2>&1"
         else
           "pdftk #{ESCAPE[pdf]} burst output #{page_path} 2>&1"
         end
         result = `#{cmd}`.chomp
-        FileUtils.rm('doc_data.txt') if File.exists?('doc_data.txt')
+        FileUtils.rm('doc_data.txt') if File.exist?('doc_data.txt')
         raise ExtractionFailed, result if $? != 0
         result
       end
diff --git a/lib/docsplit/pdf_extractor.rb b/lib/docsplit/pdf_extractor.rb
index ac487f0..a479265 100644
--- a/lib/docsplit/pdf_extractor.rb
+++ b/lib/docsplit/pdf_extractor.rb
@@ -16,14 +16,14 @@ def osx?
     def linux?
       !!HOST_OS.match(/linux/i)
     end
-    
+
     # The first line of the help output holds the name and version number
     # of the office software to be used for extraction.
     def version_string
       unless @@version_string
         null = windows? ? "NUL" : "/dev/null"
         @@version_string = `#{office_executable} -h 2>#{null}`.split("\n").first
-        if !!@@version_string.match(/[0-9]*/)
+        if !!@@version_string.to_s.match(/[0-9]*/)
           @@version_string = `#{office_executable} --version`.split("\n").first
         end
       end
@@ -35,10 +35,10 @@ def libre_office?
     def open_office?
       !!version_string.match(/^OpenOffice.org/)
     end
-    
+
     # A set of default locations to search for office software
     # These have been extracted from JODConverter.  Each listed
-    # path should contain a directory "program" which in turn 
+    # path should contain a directory "program" which in turn
     # contains the "soffice" executable.
     # see: https://github.com/mirkonasato/jodconverter/blob/master/jodconverter-core/src/main/java/org/artofsolving/jodconverter/office/OfficeUtils.java#L63-L91
     def office_search_paths
@@ -61,12 +61,15 @@ def office_search_paths
           /usr/lib64/openoffice
           /opt/openoffice.org3
           /app/vendor/libreoffice
+          /usr/bin/libreoffice
           /usr/local/bin
+          /usr/lib64/libreoffice
+          /usr/lib64/openoffice.org3
         )
       end
       search_paths
     end
-    
+
     # Identify the path to a working office executable.
     def office_executable
       paths = office_search_paths
@@ -75,10 +78,10 @@ def office_executable
       # raise an error if that path isn't valid, otherwise, add
       # it to the front of our search paths.
       if ENV['OFFICE_PATH']
-        raise ArgumentError, "No such file or directory #{ENV['OFFICE_PATH']}" unless File.exists? ENV['OFFICE_PATH']
+        raise ArgumentError, "No such file or directory #{ENV['OFFICE_PATH']}" unless File.exist? ENV['OFFICE_PATH']
         paths.unshift(ENV['OFFICE_PATH'])
       end
-      
+
       # The location of the office executable is OS dependent
       path_pieces = ["soffice"]
       if windows?
@@ -88,15 +91,15 @@ def office_executable
       else
         path_pieces += [["program", "soffice"]]
       end
-      
+
       # Search for the first suitable office executable
       # and short circuit an executable is found.
       paths.each do |path|
-        if File.exists? path
+        if File.exist? path
           @@executable ||= path unless File.directory? path
           path_pieces.each do |pieces|
             check_path = File.join(path, pieces)
-            @@executable ||= check_path if File.exists? check_path
+            @@executable ||= check_path if File.exist? check_path
           end
         end
         break if @@executable
@@ -104,16 +107,16 @@ def office_executable
       raise OfficeNotFound, "No office software found" unless @@executable
       @@executable
     end
-    
+
     # Used to specify the office location for JODConverter
     def office_path
       File.dirname(File.dirname(office_executable))
     end
-    
+
     # Convert documents to PDF.
     def extract(docs, opts)
       out = opts[:output] || '.'
-      FileUtils.mkdir_p out unless File.exists?(out)
+      FileUtils.mkdir_p out unless File.exist?(out)
       [docs].flatten.each do |doc|
         ext = File.extname(doc)
         basename = File.basename(doc, ext)
@@ -125,7 +128,7 @@ def extract(docs, opts)
           if libre_office?
             # Set the LibreOffice user profile, so that parallel uses of cloudcrowd don't trip over each other.
             ENV['SYSUSERCONFIG']="file://#{File.expand_path(escaped_out)}"
-            
+
             options = "--headless --invisible  --norestore --nolockcheck --convert-to pdf --outdir #{escaped_out} #{escaped_doc}"
             cmd = "#{office_executable} #{options} 2>&1"
             result = `#{cmd}`.chomp
@@ -144,9 +147,9 @@ def extract(docs, opts)
     LOGGING       = "-Djava.util.logging.config.file=#{ESCAPED_ROOT}/vendor/logging.properties"
 
     HEADLESS      = "-Djava.awt.headless=true"
-    
+
     private
-    
+
     # Runs a Java command, with quieted logging, and the classpath set properly.
     def run_jod(command, pdfs, opts, return_output=false)
 
diff --git a/lib/docsplit/text_extractor.rb b/lib/docsplit/text_extractor.rb
index 985abdd..f3390e8 100644
--- a/lib/docsplit/text_extractor.rb
+++ b/lib/docsplit/text_extractor.rb
@@ -28,7 +28,7 @@ def initialize
     # Extract text from a list of PDFs.
     def extract(pdfs, opts)
       extract_options opts
-      FileUtils.mkdir_p @output unless File.exists?(@output)
+      FileUtils.mkdir_p @output unless File.exist?(@output)
       [pdfs].flatten.each do |pdf|
         @pdf_name = File.basename(pdf, File.extname(pdf))
         pages = (@pages == 'all') ? 1..Docsplit.extract_length(pdf) : @pages
@@ -80,7 +80,7 @@ def extract_from_ocr(pdf, pages)
         clean_text(base_path + '.txt') if @clean_ocr
       end
     ensure
-      FileUtils.remove_entry_secure tempdir if File.exists?(tempdir)
+      FileUtils.remove_entry_secure tempdir if File.exist?(tempdir)
     end
 
 
@@ -102,17 +102,26 @@ def run(command)
       result
     end
 
+    # Run pdftotext command
+    def run_pdftotext(pdf, text_path, options=[])
+      options << '-enc UTF-8'
+      options << '-layout' if @keep_layout
+
+      run "pdftotext #{options.join(' ')} #{ESCAPE[pdf]} #{ESCAPE[text_path]} 2>&1"
+    end
+
     # Extract the full contents of a pdf as a single file, directly.
     def extract_full(pdf)
       text_path = File.join(@output, "#{@pdf_name}.txt")
-      run "pdftotext -enc UTF-8 #{ESCAPE[pdf]} #{ESCAPE[text_path]} 2>&1"
+      run_pdftotext pdf, text_path
     end
 
     # Extract the contents of a single page of text, directly, adding it to
     # the `@pages_to_ocr` list if the text length is inadequate.
     def extract_page(pdf, page)
       text_path = File.join(@output, "#{@pdf_name}_#{page}.txt")
-      run "pdftotext -enc UTF-8 -f #{page} -l #{page} #{ESCAPE[pdf]} #{ESCAPE[text_path]} 2>&1"
+      run_pdftotext pdf, text_path, ["-f #{page}", "-l #{page}"]
+
       unless @forbid_ocr
         @pages_to_ocr.push(page) if File.read(text_path).length < MIN_TEXT_PER_PAGE
       end
@@ -126,6 +135,7 @@ def extract_options(options)
       @language           = options[:language] || 'eng'
       @clean_ocr          = (!(options[:clean] == false) and @language == 'eng')
       @detect_orientation = ((options[:detect_orientation] != false) and DEPENDENCIES[:osd])
+      @keep_layout        = options.fetch(:layout, false)
     end
 
   end
diff --git a/noto_bolt.svg b/noto_bolt.svg
new file mode 100644
index 0000000..226fcdb
--- /dev/null
+++ b/noto_bolt.svg
@@ -0,0 +1,11 @@
+
+
+
+
+
+
+
diff --git a/test/test_helper.rb b/test/test_helper.rb
index 9c37b2b..2357c5a 100755
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -10,7 +10,7 @@ class Minitest::Test
   OUTPUT = 'test/output'
 
   def clear_output
-    FileUtils.rm_r(OUTPUT) if File.exists?(OUTPUT)
+    FileUtils.rm_r(OUTPUT) if File.exist?(OUTPUT)
   end
 
   def teardown