[kepler-users] Size Limitation of ArrayTokens?

[Stefan Proell]

Dear Hogan,
thanks for the hint.
I wrote my own Actor now that reads the whole file without requiring 
caching.

Greetings,
Stefan


Am 2012-05-22 16:11, schrieb Hogan, D. (GE Energy):
> The org.geon.BinaryFileReader actor has a limit of 20,000 bytes per
> iteration.  If you use that actor, you need to either process in up to
> 20,000 byte chunks or cache the results until the endOfFile port is
> true.
>
> Not all actors have this behavior.  For instance,
> ptolemy.actor.lib.io.FileReader and
> org.resurgence.actor.SimpleFileReader read the entire file as a
> character stream and output a single string.
>
> -----Original Message-----
> From: kepler-users-bounces at kepler-project.org
> [mailto:kepler-users-bounces at kepler-project.org] On Behalf Of Stefan
> Proell
> Sent: Tuesday, May 22, 2012 10:05 AM
> To: kepler-users at kepler-project.org
> Subject: Re: [kepler-users] Size Limitation of ArrayTokens?
>
> Hi!
> There seems to be a problem with the buffer in the BinaryFileReader
> Class, which is limited to 20000 bytes, as you can see in the
> corresponding source file (line 277).
>
>       /** The current bytes read. */
>       protected byte[] bytesRead = new byte[20000];
>
> It only is reading this first 20000 bytes and does not proceed after
> this. I will have a closer look at ...
>
> Stefan
>
> Am 2012-05-22 15:12, schrieb Stefan Proell:
>> Dear Matt,
>> thanks for your example. It works fine, but does not solve my problem.
>> I use a Binary File Reader to read a file from my local disk. The
>> Binary File Reader reads a local file path or URL and outputs an array
>> of bytes. No matter what file I want to read, the array size is
>> exactly 20 000 (bytes). I attached a small sample workflow to
>> demonstrate this. It simply reads a file (in my testcase this file is
>> about 40 MB) and displays the length of the resulting Byte Array.
>>
>> Do you have any idea why this behaviour occurs and can your reproduce
> it?
>> Kind regards,
>> Stefan
>>
>>
>> Am 2012-05-21 19:05, schrieb Matt Jones:
>>> I also took a look, and tried a larger array -- the attached workflow
>>> processes 30K integers through an array and back out again, and works
>>> fine.   So there isn't an inherent limit at 20K elements.  So I'm not
>>> quite sure what's happening with yours, but maybe this will help to
>>> debug.
>>>
>>> Matt
>>>
>>>
>>> On Mon, May 21, 2012 at 8:32 AM, Christopher Brooks
>>> <cxh at eecs.berkeley.edu<mailto:cxh at eecs.berkeley.edu>>  wrote:
>>>
>>>      HI Stefan,
>>>      A quick search of the web shows no apparent limits.
>>>      A more complete test case would help.
>>>
>>>      The way I would approach this is by splitting up the problem into
>>>      first just reading the data and being sure that worked and then
>>>      adding more like encoding.
>>>
>>>      _Christopher
>>>
>>>
>>>      On 5/21/12 9:20 AM, Stefan Proell wrote:
>>>
>>>          Dear Mailing list,
>>>          I wrote my own Actor which I need in order to encode binary
>>>          files via Base64 and pass the resulting encoded String to a
>>>          REST Service. I feed the binary file with a
>>>          BinaryFileReader-Actor to my encoding Actor and cast the
> Token
>>>          to an Array, as I found no other solution to receive the
> bytes
>>>          from the file. I then use a standard (and also deprecated)
>>>          method for encoding the file first to Base64 and then send it
>>>          through some URL-safe encoding. The method looks like this:
>>>
>>>
>>>          @Override
>>>             public void fire() throws IllegalActionException {
>>>
>>>                 super.fire();
>>>
>>>                 // Read file from InputPort  and convert to a
> ByteArray
>>>                 ArrayToken inputToken = (ArrayToken) inputPort.get(0);
>>>                 byte[] inputBytes =
>>>          ArrayToken.arrayTokenToUnsignedByteArray(inputToken);
>>>
>>>                 // Encoding
>>>                 String encode = new
>>>           String(Base64.encodeBase64(inputBytes));
>>>                 String encodedString =
>>> java.net.URLEncoder.encode(encode);
>>>
>>>                 output.send(0, new StringToken(encodedString));
>>>
>>>             }
>>>
>>>          My problem is that the encoded file is truncated by the actor
>>>          and hence not usable for further processing. There seems to
> be
>>>          a limit of 20 000 array elements (bytes), which is not
>>>          sufficient for my purpose. Does anyone have an idea why the
>>>          file is chopped off after 20k letters?
>>>
>>>          Thanks in advance,
>>>          Stefan
>>>
>>>          _______________________________________________
>>>          Kepler-users mailing list
>>>          Kepler-users at kepler-project.org
>>> <mailto:Kepler-users at kepler-project.org>
>>>
> http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>>>
>>>      --     Christopher Brooks, PMP                       University
>>> of California
>>>      CHESS Executive Director                      US Mail: 337 Cory
> Hall
>>>      Programmer/Analyst CHESS/Ptolemy/Trust        Berkeley, CA
>>> 94720-1774
>>>      ph: 510.643.9841<tel:510.643.9841>
>>>       (Office: 545Q Cory)
>>>      home: (F-Tu) 707.665.0131<tel:707.665.0131>  cell: 707.332.0670
>>> <tel:707.332.0670>
>>>
>>>
>>>      _______________________________________________
>>>      Kepler-users mailing list
>>>      Kepler-users at kepler-project.org
>>> <mailto:Kepler-users at kepler-project.org>
>>>      http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
>>>
>>>
>>
>>
>> _______________________________________________
>> Kepler-users mailing list
>> Kepler-users at kepler-project.org
>> http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users
> _______________________________________________
> Kepler-users mailing list
> Kepler-users at kepler-project.org
> http://lists.nceas.ucsb.edu/kepler/mailman/listinfo/kepler-users