[ome-devel] batch import into OME 5.0.2

[Josh Moore]

On Jun 23, 2014, at 9:18 PM, Yanling Liu wrote:

> Josh,

Hi Yanling,

> I have increased ulimit nofile to 65000 and the "file doesn't exist
> problem" seems to be fixed!

Glad to hear it! 65K is certainly very high, but if it gets things moving for you, all's well.

> Will try to import more images into OME...

Best of luck.
~J.

> Thanks,
> Yanling
> 
> 
> On Mon, Jun 16, 2014 at 3:55 PM, Josh Moore <josh at glencoesoftware.com>
> wrote:
> 
>> Hi Yanling,
>> 
>> my apologies for not getting to this sooner. (There are people to hit me
>> with a stick when I forget to reply to a mailing list email, but not when
>> it goes only to me)
>> 
>> On Jun 12, 2014, at 6:17 PM, Yanling Liu wrote:
>> 
>>> Josh,
>>> 
>>> bin/omero fs repos
>>> ...
>>> the repo did *not* recently move locations.
>> 
>> Yup, looks very healthy.
>> 
>> 
>>> I was trying to import a dataset including 161 folders and 8514 images. I
>>> have created a python script to iterate these 161 folders and import each
>>> folder as one dataset in OMERO.
>>> 
>>> As I indicated in previous message, I would get "file not existing error"
>>> during the import process. Also after the import process, I am having
>>> problem to do anything to OMERO including new imports and deletions. If I
>>> start new import I will get "Too many open files" error message. And
>> OMERO
>>> will not be able to accept more TCP connections.
>> 
>> Thanks for these. I'll take a look first thing tomorrow. If configuring
>> ulimit to get rid of the "Too many open files" fixes the problem, please
>> let me know.
>> 
>> Cheers,
>> ~Josh.
>> 
>> 
>> 
>>> I have attached complete import log message, along with my python script.
>>> 
>>> My python script is simply a wrapper to iteratively call omero import
>>> command.
>>> 
>>> Thanks,
>>> Yanling
>>> 
>>> 
>>> On Tue, Jun 10, 2014 at 11:51 AM, Josh Moore <josh at glencoesoftware.com>
>>> wrote:
>>> 
>>>> Hi Yanling,
>>>> 
>>>> On Jun 10, 2014, at 5:44 PM, Yanling Liu wrote:
>>>> 
>>>>> Server configuration: Ubuntu 12.04 VM with 3TB NFS storage attached.
>> OME5
>>>>> is installed and the ManagedRepository is located on the 3TB NFS
>> storage.
>>>>> 
>>>>> When using command line import tool to inject one user folder plus its
>>>>> subfolders to one dataset in OME, I have got following error on certain
>>>>> TIFF images (file path removed from log info)  :
>>>> 
>>>> Would you mind sending the full log (via private email if you'd prefer)?
>>>> Also having the command-line as you executed it would be useful.
>>>> 
>>>> 
>>>>> I have verified the file 51-11.tif exists as
>>>>> ManagedRepository/acharyaj_52/2014-06/06/16-43-40.589/51-11.tif. Tried
>>>> run
>>>>> tiffinfo tool again the image file and found no problem. Don't know why
>>>> OME
>>>>> report an existing file as non-exist...
>>>> 
>>>> Could you send the output of:
>>>> 
>>>> bin/omero fs repos
>>>> 
>>>> and:
>>>> 
>>>> bin/omero config get | grep omero.data.dir
>>>> 
>>>> Did the repo recently move locations?
>>>> 
>>>> 
>>>>> Thanks,
>>>>> Yanling
>>>> 
>>>> Cheers,
>>>> ~Josh
>>> <output.txt><importSingleUserImages_s.py>
>> 
>>