Errors and warnings while fragment building

[elhossary]

Using READemption v2.0.3
Besides the very long time it takes to build fragments, I have warnings because of the missing index files for bam files. Then errors occur due to "Too many open files".

        total   mapped  (%)     unique  (%)     multi   (%)     split   (%)                                                                       
all     45714906        44783193        97.96%  4687693 10.25%  40095500        87.71%  0       0.00%                                             
pair    22857453        21504215        94.08%  2033415 8.90%   19470800        85.18%  0       0.00%                                             
[SEGEMEHL] Tue Jul 11 12:34:39 2023:                                                                                                              
Goodbye.                                                                                                                                          
 "Die Ficker!" (Thommy)                                                                                                                           
controller align_paired_end_reads stop 2023-07-11 13:51:57.318376                                                                                 
controller generate_read_alignment_stats start 2023-07-11 13:51:57.318614                                                                         
readalignerstats count_aligned_reads_and_alignments start 2023-07-11 13:51:57.764202                                                              
readalignerstats count_aligned_reads_and_alignments start 2023-07-11 13:51:57.764462                                                              
readalignerstats count_aligned_reads_and_alignments start 2023-07-11 13:51:57.764490                                                              
readalignerstats count_aligned_reads_and_alignments start 2023-07-11 13:51:57.768954                                                              
readalignerstats count_aligned_reads_and_alignments stop 2023-07-12 13:42:39.552076                                                               
readalignerstats count_unaligned_reads start 2023-07-12 13:42:39.610802
readalignerstats count_unaligned_reads stop 2023-07-12 13:42:40.274719
readalignerstats count_aligned_reads_and_alignments stop 2023-07-12 16:58:52.556453
readalignerstats count_unaligned_reads start 2023-07-12 16:58:52.559282                                                                           
readalignerstats count_unaligned_reads stop 2023-07-12 16:58:54.523031                                                                            
readalignerstats count_aligned_reads_and_alignments stop 2023-07-12 18:13:50.040636                                                               
readalignerstats count_unaligned_reads start 2023-07-12 18:13:50.042781                                                                           
readalignerstats count_unaligned_reads stop 2023-07-12 18:13:51.502212                                                                            
readalignerstats count_aligned_reads_and_alignments stop 2023-07-12 18:39:25.329928                                                               
readalignerstats count_unaligned_reads start 2023-07-12 18:39:25.332696                                                                           
readalignerstats count_unaligned_reads stop 2023-07-12 18:39:26.450343                                                                            
controller generate_read_alignment_stats stop 2023-07-12 18:39:27.672006                                                                          
controller sort bams by name and index start 2023-07-12 19:35:14.979915                                                                           
controller sort bams by name and index end 2023-07-12 21:12:23.925582                                                                             
controller build_fragments start 2023-07-12 21:12:23.925959                                                                                       
[E::idx_find_and_load] [E::idx_find_and_load] [E::idx_find_and_load] Could not retrieve index file for 'READemption_analysis_NEBNext-Ultra-II_kit/
output/align/alignments/NEBNext-Ultra-II_kit_RNA-Seq_OD_0.2_rep1_alignments_final_sorted.bam'Could not retrieve index file for 'READemption_analys
is_NEBNext-Ultra-II_kit/output/align/alignments/NEBNext-Ultra-II_kit_RNA-Seq_OD_2.0_rep2_alignments_final_sorted.bam'Could not retrieve index file
 for 'READemption_analysis_NEBNext-Ultra-II_kit/output/align/alignments/NEBNext-Ultra-II_kit_RNA-Seq_OD_2.0_rep1_alignments_final_sorted.bam'     
[E::idx_find_and_load] Could not retrieve index file for 'READemption_analysis_NEBNext-Ultra-II_kit/output/align/alignments/NEBNext-Ultra-II_kit_R
NA-Seq_OD_0.2_rep2_alignments_final_sorted.bam'                                                                                                   
[E::hts_open_format] Failed to open file "READemption_analysis_NEBNext-Ultra-II_kit/output/align/alignments/NEBNext-Ultra-II_kit_RNA-Seq_OD_0.2_re
p1_alignments_final_fragments.bam_sorted.tmp.1009.bam" : Too many open files                                                                      
[E::hts_open_format] Failed to open file "READemption_analysis_NEBNext-Ultra-II_kit/output/align/alignments/NEBNext-Ultra-II_kit_RNA-Seq_OD_2.0_re
p2_alignments_final_fragments.bam_sorted.tmp.1005.bam" : Too many open files                                                                      
[E::hts_open_format] Failed to open file "READemption_analysis_NEBNext-Ultra-II_kit/output/align/alignments/NEBNext-Ultra-II_kit_RNA-Seq_OD_0.2_re
p2_alignments_final_fragments.bam_sorted.tmp.1007.bam" : Too many open files                                                                      
[E::hts_open_format] Failed to open file "READemption_analysis_NEBNext-Ultra-II_kit/output/align/alignments/NEBNext-Ultra-II_kit_RNA-Seq_OD_2.0_re
p1_alignments_final_fragments.bam_sorted.tmp.1003.bam" : Too many open files
concurrent.futures.process._RemoteTraceback:                                                                                                      
"""                                                                                                                                               
Traceback (most recent call last):                                                                                                                
  File "/usr/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker                                                          
    r = call_item.fn(*call_item.args, **call_item.kwargs)                                                                                         
  File "/usr/local/lib/python3.10/dist-packages/reademptionlib/fragmentbuilder.py", line 113, in build_bam_file_with_fragments                    
    pysam.sort("-o", tmp_sorted_outfile, fragment_alignment_path)                                                                                 
  File "/home/muhammad/.local/lib/python3.10/site-packages/pysam/utils.py", line 69, in __call__                                                  
    raise SamtoolsError(                                                                                                                          
pysam.utils.SamtoolsError: 'samtools returned with error 1: stdout=, stderr=[bam_sort_core] merging from 6420 files and 1 in-memory blocks...\nsam
tools sort: fail to open "READemption_analysis_NEBNext-Ultra-II_kit/output/align/alignments/NEBNext-Ultra-II_kit_RNA-Seq_OD_0.2_rep2_alignments_fi
nal_fragments.bam_sorted.tmp.1007.bam": Too many open files\n'                                                                                    
"""                                                                                                                                               

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/reademption", line 715, in <module>
    main()
  File "/usr/local/bin/reademption", line 22, in main
    args.func(controller)
  File "/usr/local/bin/reademption", line 687, in align_reads
    controller.align_reads()
  File "/usr/local/lib/python3.10/dist-packages/reademptionlib/controller.py", line 212, in align_reads
    self._build_fragments()
  File "/usr/local/lib/python3.10/dist-packages/reademptionlib/controller.py", line 298, in _build_fragments
    self._check_job_completeness(jobs)
  File "/usr/local/lib/python3.10/dist-packages/reademptionlib/controller.py", line 801, in _check_job_completeness
    raise (job.exception())
pysam.utils.SamtoolsError: 'samtools returned with error 1: stdout=, stderr=[bam_sort_core] merging from 6420 files and 1 in-memory blocks...\nsam
tools sort: fail to open "READemption_analysis_NEBNext-Ultra-II_kit/output/align/alignments/NEBNext-Ultra-II_kit_RNA-Seq_OD_0.2_rep2_alignments_fi
nal_fragments.bam_sorted.tmp.1007.bam": Too many open files\n'