[kepler-users] Exec Component Binary Data

[Christopher Brooks]

I have not thought that hard about how to pass binary data, probably the 
thing to do would be to enhance the Exec actor to optionally pass an 
ObjectToken that would contain the binary data.

Offhand, I don't know of anything in the StringToken that should prevent 
binary data from being used, but there could be issues. Some simple test 
cases would help here.

One thing is that reading data from files can result in transformation 
of the data.  ptolemy/util/FileUtilities.java has a method that safely 
reads a file and outputs it using ByteArrayOutputStream.

BTW - I'm used to Unix passing data as files that have newline 
characters and using tools and scripts on textual data.  A few 
line-based tools tend to fail if the file does not have at least one 
newline at the end of the file.  Interestingly, the find command can 
generate an output that has file names separated by null characters and 
xargs can process that data with the -0 option.  For example, these 
commands find file names with spaces in them and then call xargs to run 
a command.

find . -name "* *" -print0 | xargs -0 ls -l

(Of course, one could just do find . -name "* *" -ls, but the point is 
to show how to pass filenames with spaces to xargs)

On 4/19/13 6:06 PM, Jianwu Wang wrote:
> Hi David,
>
> External execution assume the data for output port is string and 
> process it accordingly. So I don't think it can correctly send binary 
> data from its output port.
>
> One way workaround is to redirect the binary output into a file and 
> the next 'External execution' actor will read data from the file. If 
> it fits your needs, I can send a demo workflow for it.
> Best wishes
>
> Sincerely yours
>
> Jianwu Wang, Ph.D.
> jianwu at sdsc.edu
> http://users.sdsc.edu/~jianwu/
>
> Assistant Project Scientist
> Scientific Workflow Automation Technologies (SWAT) Laboratory
> San Diego Supercomputer Center
> University of California, San Diego
> San Diego, CA, U.S.A.
> On 4/18/13 11:02 AM, Brown, David M JR wrote:
>> To whom it may concern,
>> I've got a set of binary tools that use standard unix style pipes to 
>> communicate and process data. So to write this up in Kepler I chose 
>> to use the external execution component and run each command 
>> separately connecting one components standard output to the other 
>> components standard input. However, the communication isn't in text 
>> its binary. I'm having issues where the first part of the process 
>> doesn't recognize the header sent to the program as valid.
>> Are there any issues with sending binary data from external execution 
>> components output port?
>> Thanks,
>> - David Brown
>>
>>
>> _______________________________________________
>> Kepler-users mailing list
>> Kepler-users at kepler-project.org
>> http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>
>
>
> _______________________________________________
> Kepler-users mailing list
> Kepler-users at kepler-project.org
> http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users

-- 
Christopher Brooks, PMP                       University of California
Academic Program Manager & Software Engineer  US Mail: 337 Cory Hall
CHESS/iCyPhy/Ptolemy/TerraSwarm               Berkeley, CA 94720-1774
cxh at eecs.berkeley.edu, 707.332.0670           (Office: 545Q Cory)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nceas.ucsb.edu/kepler/pipermail/kepler-users/attachments/20130422/12081b4e/attachment.html>